[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 08:07:40.878947 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:38129,127.28.55.125:33403,127.28.55.124:42093
I20240503 08:07:40.879263 28893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:07:40.879417 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:40.883567 28898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.883576 28901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:40.883529 28893 server_base.cc:1034] running on GCE node
W20240503 08:07:40.883556 28899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:40.883966 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:40.884007 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:40.884033 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723660884033 us; error 0 us; skew 500 ppm
I20240503 08:07:40.885115 28893 webserver.cc:460] Webserver started at http://127.28.55.126:41791/ using document root <none> and password file <none>
I20240503 08:07:40.885262 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:40.885301 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:40.885363 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:40.886509 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "499b4241406248b198b19c38cc0436f9"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-mlgg"
I20240503 08:07:40.887773 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:07:40.888298 28906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.888460 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:40.888525 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "499b4241406248b198b19c38cc0436f9"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-mlgg"
I20240503 08:07:40.888576 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:40.909767 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:40.910054 28893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:07:40.910151 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:40.914088 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:38129
I20240503 08:07:40.914148 28958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:38129 every 8 connection(s)
I20240503 08:07:40.914528 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:07:40.914875 28959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:07:40.915791 28962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.915851 28964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.915973 28961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:40.916224 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:40.916342 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:40.916374 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:40.916389 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723660916390 us; error 0 us; skew 500 ppm
I20240503 08:07:40.917064 28893 webserver.cc:460] Webserver started at http://127.28.55.125:39901/ using document root <none> and password file <none>
I20240503 08:07:40.917150 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:40.917203 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:40.917253 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:40.917354 28959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:40.917492 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root/instance:
uuid: "f09fa2338d734346ba0a50c8b5125131"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-mlgg"
I20240503 08:07:40.918388 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:40.918860 28971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.918995 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.919041 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root
uuid: "f09fa2338d734346ba0a50c8b5125131"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-mlgg"
I20240503 08:07:40.919086 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:40.922767 28959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:40.923123 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20240503 08:07:40.923136 28907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.125:33403: connect: Connection refused (error 111)
I20240503 08:07:40.923391 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240503 08:07:40.923585 28959 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.28.55.125:33403: Network error: Client connection negotiation failed: client connection to 127.28.55.125:33403: connect: Connection refused (error 111)
I20240503 08:07:40.926651 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.125:33403
I20240503 08:07:40.926709 29024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.125:33403 every 8 connection(s)
I20240503 08:07:40.926970 29025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:40.926975 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:07:40.927881 29025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:07:40.928189 29027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.928251 29028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.928444 29030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:40.928885 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:40.929000 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:40.929035 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:40.929049 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723660929049 us; error 0 us; skew 500 ppm
I20240503 08:07:40.929701 28893 webserver.cc:460] Webserver started at http://127.28.55.124:44125/ using document root <none> and password file <none>
I20240503 08:07:40.929770 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:40.929811 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:40.929857 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:40.930099 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root/instance:
uuid: "e329d9de4de546b9a45221503ed371da"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-mlgg"
I20240503 08:07:40.931067 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:40.931121 29025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:40.931576 29037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.931740 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.931783 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root
uuid: "e329d9de4de546b9a45221503ed371da"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-mlgg"
I20240503 08:07:40.931823 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:40.934235 29025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:07:40.934895 29025 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.28.55.124:42093: Network error: Client connection negotiation failed: client connection to 127.28.55.124:42093: connect: Connection refused (error 111)
I20240503 08:07:40.937381 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:40.937616 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:40.940814 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.124:42093
I20240503 08:07:40.940869 29090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.124:42093 every 8 connection(s)
I20240503 08:07:40.941009 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:07:40.941105 29091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:40.941780 29091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:40.944633 29091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:40.947032 29091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:40.950649 29091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da: Bootstrap starting.
I20240503 08:07:40.951306 29091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:40.951581 29091 log.cc:824] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da: Log is configured to *not* fsync() on all Append() calls
I20240503 08:07:40.952188 29091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da: No bootstrap required, opened a new log
I20240503 08:07:40.953522 29091 raft_consensus.cc:348] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } }
I20240503 08:07:40.953640 29091 raft_consensus.cc:374] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:40.953657 29091 raft_consensus.cc:729] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e329d9de4de546b9a45221503ed371da, State: Initialized, Role: FOLLOWER
I20240503 08:07:40.953759 29091 consensus_queue.cc:260] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } }
I20240503 08:07:40.954033 29091 sys_catalog.cc:564] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:40.953994 29096 sys_catalog.cc:455] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:40.954088 29096 sys_catalog.cc:458] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [sys.catalog]: This master's current role is: FOLLOWER
W20240503 08:07:40.955577 29107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:07:40.955623 29107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:07:40.972765 28959 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } attempt: 1
I20240503 08:07:40.975965 28959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:40.979466 28959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9: Bootstrap starting.
I20240503 08:07:40.979862 28959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:40.980453 28959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9: No bootstrap required, opened a new log
I20240503 08:07:40.980613 28959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } }
I20240503 08:07:40.980675 28959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:40.980693 28959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499b4241406248b198b19c38cc0436f9, State: Initialized, Role: FOLLOWER
I20240503 08:07:40.980753 28959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } }
I20240503 08:07:40.980949 28959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:40.980974 29110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:40.981072 29110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:40.982148 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240503 08:07:40.982290 29121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:07:40.982338 29121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:07:40.987046 29025 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } attempt: 1
I20240503 08:07:40.991115 29025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131: Bootstrap starting.
I20240503 08:07:40.991591 29025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:40.992308 29025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131: No bootstrap required, opened a new log
I20240503 08:07:40.992491 29025 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } }
I20240503 08:07:40.992588 29025 raft_consensus.cc:374] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:40.992609 29025 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09fa2338d734346ba0a50c8b5125131, State: Initialized, Role: FOLLOWER
I20240503 08:07:40.992666 29025 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } }
I20240503 08:07:40.992858 29025 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:40.992866 29123 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:40.992980 29123 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:40.994338 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240503 08:07:40.994697 29134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240503 08:07:40.994716 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:40.994778 29134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 08:07:40.996228 29135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.996302 29138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.996245 29136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:40.996559 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:40.996673 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:40.996712 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:40.996729 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723660996729 us; error 0 us; skew 500 ppm
I20240503 08:07:40.997496 28893 webserver.cc:460] Webserver started at http://127.28.55.65:33051/ using document root <none> and password file <none>
I20240503 08:07:40.997576 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:40.997624 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:40.997674 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:40.997970 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-mlgg"
I20240503 08:07:40.998761 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:40.999270 29143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.999410 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.999451 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-mlgg"
I20240503 08:07:40.999496 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:41.004755 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:41.005003 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:41.005210 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:41.005509 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:41.005546 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:41.005565 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:41.005578 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:41.009084 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:35631
I20240503 08:07:41.009243 29206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:35631 every 8 connection(s)
I20240503 08:07:41.009585 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:41.010768 29212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:41.012876 29207 heartbeater.cc:347] Connected to a master server at 127.28.55.124:42093
I20240503 08:07:41.012952 29207 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:41.013172 29207 heartbeater.cc:511] Master 127.28.55.124:42093 requested a full tablet report, sending...
I20240503 08:07:41.013695 29055 ts_manager.cc:194] Registered new tserver with Master: 8a79d7e5f8254c71b41b9d6d89e60fd0 (127.28.55.65:35631)
W20240503 08:07:41.016095 29215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:41.016108 29213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:41.016296 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:41.016505 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:41.016562 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:41.016606 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723661016606 us; error 0 us; skew 500 ppm
I20240503 08:07:41.017468 28893 webserver.cc:460] Webserver started at http://127.28.55.66:37247/ using document root <none> and password file <none>
I20240503 08:07:41.017593 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:41.017668 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:41.017737 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:41.018105 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "5df5736f58de4bfc9ecaf8ec39e2be2e"
format_stamp: "Formatted at 2024-05-03 08:07:41 on dist-test-slave-mlgg"
I20240503 08:07:41.018458 29208 heartbeater.cc:347] Connected to a master server at 127.28.55.126:38129
I20240503 08:07:41.018510 29208 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:41.018610 29208 heartbeater.cc:511] Master 127.28.55.126:38129 requested a full tablet report, sending...
I20240503 08:07:41.018936 28923 ts_manager.cc:194] Registered new tserver with Master: 8a79d7e5f8254c71b41b9d6d89e60fd0 (127.28.55.65:35631)
I20240503 08:07:41.019353 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:41.019527 29209 heartbeater.cc:347] Connected to a master server at 127.28.55.125:33403
I20240503 08:07:41.019574 29209 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:41.019673 29209 heartbeater.cc:511] Master 127.28.55.125:33403 requested a full tablet report, sending...
I20240503 08:07:41.019941 29221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:41.019995 28989 ts_manager.cc:194] Registered new tserver with Master: 8a79d7e5f8254c71b41b9d6d89e60fd0 (127.28.55.65:35631)
I20240503 08:07:41.020149 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:07:41.020195 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "5df5736f58de4bfc9ecaf8ec39e2be2e"
format_stamp: "Formatted at 2024-05-03 08:07:41 on dist-test-slave-mlgg"
I20240503 08:07:41.020238 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:41.035262 29096 raft_consensus.cc:482] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:41.035391 29096 raft_consensus.cc:504] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } }
I20240503 08:07:41.035678 29096 leader_election.cc:288] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 499b4241406248b198b19c38cc0436f9 (127.28.55.126:38129), f09fa2338d734346ba0a50c8b5125131 (127.28.55.125:33403)
I20240503 08:07:41.035953 28999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e329d9de4de546b9a45221503ed371da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09fa2338d734346ba0a50c8b5125131" is_pre_election: true
I20240503 08:07:41.035950 28933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e329d9de4de546b9a45221503ed371da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499b4241406248b198b19c38cc0436f9" is_pre_election: true
I20240503 08:07:41.036084 28999 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e329d9de4de546b9a45221503ed371da in term 0.
I20240503 08:07:41.036098 28933 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e329d9de4de546b9a45221503ed371da in term 0.
I20240503 08:07:41.036247 29038 leader_election.cc:302] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e329d9de4de546b9a45221503ed371da, f09fa2338d734346ba0a50c8b5125131; no voters:
I20240503 08:07:41.036402 29096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:41.036438 29096 raft_consensus.cc:482] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:41.036456 29096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:41.037128 29096 raft_consensus.cc:504] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } }
I20240503 08:07:41.037263 29096 leader_election.cc:288] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [CANDIDATE]: Term 1 election: Requested vote from peers 499b4241406248b198b19c38cc0436f9 (127.28.55.126:38129), f09fa2338d734346ba0a50c8b5125131 (127.28.55.125:33403)
I20240503 08:07:41.037439 28999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e329d9de4de546b9a45221503ed371da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09fa2338d734346ba0a50c8b5125131"
I20240503 08:07:41.037464 28933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e329d9de4de546b9a45221503ed371da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499b4241406248b198b19c38cc0436f9"
I20240503 08:07:41.037500 28999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:41.037503 28933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:41.038187 28933 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e329d9de4de546b9a45221503ed371da in term 1.
I20240503 08:07:41.038187 28999 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e329d9de4de546b9a45221503ed371da in term 1.
I20240503 08:07:41.038383 29041 leader_election.cc:302] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499b4241406248b198b19c38cc0436f9, e329d9de4de546b9a45221503ed371da; no voters:
I20240503 08:07:41.038452 29096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:41.038609 29096 raft_consensus.cc:686] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 1 LEADER]: Becoming Leader. State: Replica: e329d9de4de546b9a45221503ed371da, State: Running, Role: LEADER
I20240503 08:07:41.038697 29096 consensus_queue.cc:237] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } }
I20240503 08:07:41.039197 29226 sys_catalog.cc:455] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [sys.catalog]: SysCatalogTable state changed. Reason: New leader e329d9de4de546b9a45221503ed371da. Latest consensus state: current_term: 1 leader_uuid: "e329d9de4de546b9a45221503ed371da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:41.039280 29226 sys_catalog.cc:458] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:41.039456 29228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:07:41.039651 29228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:07:41.040956 28933 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [term 1 FOLLOWER]: Refusing update from remote peer e329d9de4de546b9a45221503ed371da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:41.040956 28999 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [term 1 FOLLOWER]: Refusing update from remote peer e329d9de4de546b9a45221503ed371da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:41.041235 29226 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [LEADER]: Connected to new peer: Peer: permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:41.041317 29096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:41.041937 29123 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e329d9de4de546b9a45221503ed371da. Latest consensus state: current_term: 1 leader_uuid: "e329d9de4de546b9a45221503ed371da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:41.042043 29123 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:41.041937 29110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e329d9de4de546b9a45221503ed371da. Latest consensus state: current_term: 1 leader_uuid: "e329d9de4de546b9a45221503ed371da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:41.042114 29110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:41.042681 29226 sys_catalog.cc:455] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e329d9de4de546b9a45221503ed371da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:41.042752 29226 sys_catalog.cc:458] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:41.042790 29226 sys_catalog.cc:455] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e329d9de4de546b9a45221503ed371da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:41.042876 29226 sys_catalog.cc:458] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:41.043053 29123 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e329d9de4de546b9a45221503ed371da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:41.043112 29123 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:41.043155 29110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e329d9de4de546b9a45221503ed371da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499b4241406248b198b19c38cc0436f9" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 38129 } } peers { permanent_uuid: "f09fa2338d734346ba0a50c8b5125131" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 33403 } } peers { permanent_uuid: "e329d9de4de546b9a45221503ed371da" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 42093 } } }
I20240503 08:07:41.043193 29110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:41.043432 29230 mvcc.cc:205] Tried to move back new op lower bound from 7023508115622027264 to 7023508115615432704. Current Snapshot: MvccSnapshot[applied={T|T < 7023508115622027264}]
I20240503 08:07:41.043823 29228 catalog_manager.cc:1347] Generated new cluster ID: eb49f6e3d50f4cb9a3ced9dcf1cf707d
I20240503 08:07:41.043874 29228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:07:41.052488 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:41.052767 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:41.052961 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:41.053283 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:41.053359 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:41.053426 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:41.053472 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:41.057840 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:36161
I20240503 08:07:41.058385 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:07:41.061874 29296 heartbeater.cc:347] Connected to a master server at 127.28.55.124:42093
I20240503 08:07:41.061946 29296 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:41.062049 29296 heartbeater.cc:511] Master 127.28.55.124:42093 requested a full tablet report, sending...
I20240503 08:07:41.062284 29055 ts_manager.cc:194] Registered new tserver with Master: 5df5736f58de4bfc9ecaf8ec39e2be2e (127.28.55.66:36161)
I20240503 08:07:41.062985 29295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:36161 every 8 connection(s)
W20240503 08:07:41.064548 29302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:41.064684 29304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:41.065101 29301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:41.065724 29228 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:07:41.066371 29228 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:07:41.068285 29297 heartbeater.cc:347] Connected to a master server at 127.28.55.126:38129
I20240503 08:07:41.068396 29297 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:41.068580 29297 heartbeater.cc:511] Master 127.28.55.126:38129 requested a full tablet report, sending...
I20240503 08:07:41.069171 28923 ts_manager.cc:194] Registered new tserver with Master: 5df5736f58de4bfc9ecaf8ec39e2be2e (127.28.55.66:36161)
I20240503 08:07:41.070083 29298 heartbeater.cc:347] Connected to a master server at 127.28.55.125:33403
I20240503 08:07:41.070199 29298 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:41.070351 29298 heartbeater.cc:511] Master 127.28.55.125:33403 requested a full tablet report, sending...
I20240503 08:07:41.070668 28989 ts_manager.cc:194] Registered new tserver with Master: 5df5736f58de4bfc9ecaf8ec39e2be2e (127.28.55.66:36161)
I20240503 08:07:41.071848 29228 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da: Generated new TSK 0
I20240503 08:07:41.071916 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:41.072101 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 08:07:41.072108 29228 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 08:07:41.072144 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:41.072275 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723661072275 us; error 0 us; skew 500 ppm
I20240503 08:07:41.073115 28893 webserver.cc:460] Webserver started at http://127.28.55.67:43347/ using document root <none> and password file <none>
I20240503 08:07:41.073200 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:41.073256 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:41.073305 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:41.073591 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "54e1bcdd99bc413f9212fe3606983b88"
format_stamp: "Formatted at 2024-05-03 08:07:41 on dist-test-slave-mlgg"
I20240503 08:07:41.074443 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:41.074939 29311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:41.075089 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:41.075130 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "54e1bcdd99bc413f9212fe3606983b88"
format_stamp: "Formatted at 2024-05-03 08:07:41 on dist-test-slave-mlgg"
I20240503 08:07:41.075177 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:41.086755 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:41.087054 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:41.087395 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:41.087630 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:41.087661 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:41.087692 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:41.087723 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:41.091579 29374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:41811 every 8 connection(s)
I20240503 08:07:41.091558 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:41811
I20240503 08:07:41.095618 29376 heartbeater.cc:347] Connected to a master server at 127.28.55.126:38129
I20240503 08:07:41.095700 29376 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:41.095811 29376 heartbeater.cc:511] Master 127.28.55.126:38129 requested a full tablet report, sending...
I20240503 08:07:41.096694 28923 ts_manager.cc:194] Registered new tserver with Master: 54e1bcdd99bc413f9212fe3606983b88 (127.28.55.67:41811)
I20240503 08:07:41.098464 29375 heartbeater.cc:347] Connected to a master server at 127.28.55.124:42093
I20240503 08:07:41.098536 29375 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:41.098636 29375 heartbeater.cc:511] Master 127.28.55.124:42093 requested a full tablet report, sending...
I20240503 08:07:41.098907 29377 heartbeater.cc:347] Connected to a master server at 127.28.55.125:33403
I20240503 08:07:41.098944 29377 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:41.098958 29055 ts_manager.cc:194] Registered new tserver with Master: 54e1bcdd99bc413f9212fe3606983b88 (127.28.55.67:41811)
I20240503 08:07:41.099059 29377 heartbeater.cc:511] Master 127.28.55.125:33403 requested a full tablet report, sending...
I20240503 08:07:41.099953 29055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42754
I20240503 08:07:41.100414 28989 ts_manager.cc:194] Registered new tserver with Master: 54e1bcdd99bc413f9212fe3606983b88 (127.28.55.67:41811)
I20240503 08:07:41.100903 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008978706s
I20240503 08:07:41.982733 29121 catalog_manager.cc:1259] Loaded cluster ID: eb49f6e3d50f4cb9a3ced9dcf1cf707d
I20240503 08:07:41.982839 29121 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9: loading cluster ID for follower catalog manager: success
I20240503 08:07:41.983623 29121 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9: acquiring CA information for follower catalog manager: success
I20240503 08:07:41.983834 29121 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:07:41.995023 29134 catalog_manager.cc:1259] Loaded cluster ID: eb49f6e3d50f4cb9a3ced9dcf1cf707d
I20240503 08:07:41.995103 29134 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131: loading cluster ID for follower catalog manager: success
I20240503 08:07:41.995887 29134 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131: acquiring CA information for follower catalog manager: success
I20240503 08:07:41.996049 29134 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:07:42.015123 29055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42734
I20240503 08:07:42.063536 29055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42742
I20240503 08:07:42.100836 29375 heartbeater.cc:503] Master 127.28.55.124:42093 was elected leader, sending a full tablet report...
I20240503 08:07:42.955291 29106 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:07:43.015758 29207 heartbeater.cc:503] Master 127.28.55.124:42093 was elected leader, sending a full tablet report...
I20240503 08:07:43.064184 29296 heartbeater.cc:503] Master 127.28.55.124:42093 was elected leader, sending a full tablet report...
I20240503 08:07:43.152431 28893 test_util.cc:268] Using random seed: 919897383
I20240503 08:07:43.176703 29055 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42758:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 08:07:43.184036 29260 tablet_service.cc:1467] Processing CreateTablet for tablet 590dc86ae5794cb78377fcafeaff386f (DEFAULT_TABLE table=test-workload [id=db197500d7fe44e1ab5efd0314cf8833]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:07:43.184125 29170 tablet_service.cc:1467] Processing CreateTablet for tablet 413dc32531c849599cfe144d8fa66368 (DEFAULT_TABLE table=test-workload [id=db197500d7fe44e1ab5efd0314cf8833]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:07:43.184267 29170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413dc32531c849599cfe144d8fa66368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:43.184271 29260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590dc86ae5794cb78377fcafeaff386f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:43.184325 29339 tablet_service.cc:1467] Processing CreateTablet for tablet 59853a8d2bf94df18653db72858933df (DEFAULT_TABLE table=test-workload [id=db197500d7fe44e1ab5efd0314cf8833]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:07:43.184408 29339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59853a8d2bf94df18653db72858933df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:43.185643 29412 tablet_bootstrap.cc:492] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88: Bootstrap starting.
I20240503 08:07:43.184125 29171 tablet_service.cc:1467] Processing CreateTablet for tablet bf70b97dd01f4fe09cf9b43383db533a (DEFAULT_TABLE table=test-workload [id=db197500d7fe44e1ab5efd0314cf8833]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:07:43.185827 29413 tablet_bootstrap.cc:492] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0: Bootstrap starting.
I20240503 08:07:43.185869 29171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf70b97dd01f4fe09cf9b43383db533a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:43.185920 29414 tablet_bootstrap.cc:492] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e: Bootstrap starting.
I20240503 08:07:43.186239 29412 tablet_bootstrap.cc:654] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:43.186239 29414 tablet_bootstrap.cc:654] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:43.186260 29413 tablet_bootstrap.cc:654] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:43.186975 29413 tablet_bootstrap.cc:492] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0: No bootstrap required, opened a new log
I20240503 08:07:43.187031 29413 ts_tablet_manager.cc:1367] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:43.187098 29414 tablet_bootstrap.cc:492] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e: No bootstrap required, opened a new log
I20240503 08:07:43.187150 29414 ts_tablet_manager.cc:1367] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:43.187168 29413 raft_consensus.cc:348] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } }
I20240503 08:07:43.187244 29413 raft_consensus.cc:374] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:43.187355 29413 raft_consensus.cc:729] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a79d7e5f8254c71b41b9d6d89e60fd0, State: Initialized, Role: FOLLOWER
I20240503 08:07:43.187284 29414 raft_consensus.cc:348] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df5736f58de4bfc9ecaf8ec39e2be2e" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36161 } }
I20240503 08:07:43.187377 29414 raft_consensus.cc:374] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:43.187397 29414 raft_consensus.cc:729] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df5736f58de4bfc9ecaf8ec39e2be2e, State: Initialized, Role: FOLLOWER
I20240503 08:07:43.187403 29413 consensus_queue.cc:260] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } }
I20240503 08:07:43.187440 29413 raft_consensus.cc:388] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:43.187429 29414 consensus_queue.cc:260] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df5736f58de4bfc9ecaf8ec39e2be2e" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36161 } }
I20240503 08:07:43.187232 29412 tablet_bootstrap.cc:492] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88: No bootstrap required, opened a new log
I20240503 08:07:43.187469 29414 raft_consensus.cc:388] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:43.187500 29414 raft_consensus.cc:482] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:43.187500 29412 ts_tablet_manager.cc:1367] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:07:43.187546 29414 raft_consensus.cc:3037] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:43.187640 29412 raft_consensus.cc:348] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e1bcdd99bc413f9212fe3606983b88" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41811 } }
I20240503 08:07:43.187691 29412 raft_consensus.cc:374] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:43.187707 29412 raft_consensus.cc:729] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e1bcdd99bc413f9212fe3606983b88, State: Initialized, Role: FOLLOWER
I20240503 08:07:43.187749 29412 consensus_queue.cc:260] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e1bcdd99bc413f9212fe3606983b88" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41811 } }
I20240503 08:07:43.187791 29412 raft_consensus.cc:388] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:43.187808 29412 raft_consensus.cc:482] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:43.187822 29412 raft_consensus.cc:3037] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:43.188162 29414 raft_consensus.cc:504] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df5736f58de4bfc9ecaf8ec39e2be2e" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36161 } }
I20240503 08:07:43.188241 29414 leader_election.cc:302] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5df5736f58de4bfc9ecaf8ec39e2be2e; no voters:
I20240503 08:07:43.188329 29412 raft_consensus.cc:504] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e1bcdd99bc413f9212fe3606983b88" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41811 } }
I20240503 08:07:43.188364 29414 leader_election.cc:288] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:43.188391 29412 leader_election.cc:302] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54e1bcdd99bc413f9212fe3606983b88; no voters:
I20240503 08:07:43.188405 29418 raft_consensus.cc:2781] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:43.188475 29412 leader_election.cc:288] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:43.188498 29414 ts_tablet_manager.cc:1398] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:43.188540 29418 raft_consensus.cc:686] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 1 LEADER]: Becoming Leader. State: Replica: 5df5736f58de4bfc9ecaf8ec39e2be2e, State: Running, Role: LEADER
I20240503 08:07:43.188585 29412 ts_tablet_manager.cc:1398] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:43.188594 29418 consensus_queue.cc:237] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df5736f58de4bfc9ecaf8ec39e2be2e" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36161 } }
I20240503 08:07:43.187463 29413 raft_consensus.cc:482] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:43.188887 29413 raft_consensus.cc:3037] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:43.189373 29054 catalog_manager.cc:5484] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5df5736f58de4bfc9ecaf8ec39e2be2e (127.28.55.66). New cstate: current_term: 1 leader_uuid: "5df5736f58de4bfc9ecaf8ec39e2be2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df5736f58de4bfc9ecaf8ec39e2be2e" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36161 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:43.189472 29413 raft_consensus.cc:504] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } }
I20240503 08:07:43.189532 29413 leader_election.cc:302] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a79d7e5f8254c71b41b9d6d89e60fd0; no voters:
I20240503 08:07:43.189641 29413 leader_election.cc:288] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:43.189772 29413 ts_tablet_manager.cc:1398] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:07:43.189667 29425 raft_consensus.cc:2781] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:43.190019 29413 tablet_bootstrap.cc:492] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0: Bootstrap starting.
I20240503 08:07:43.190055 29425 raft_consensus.cc:686] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 LEADER]: Becoming Leader. State: Replica: 8a79d7e5f8254c71b41b9d6d89e60fd0, State: Running, Role: LEADER
I20240503 08:07:43.190320 29425 consensus_queue.cc:237] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } }
I20240503 08:07:43.190937 29413 tablet_bootstrap.cc:654] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:43.190907 29054 catalog_manager.cc:5484] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a79d7e5f8254c71b41b9d6d89e60fd0 (127.28.55.65). New cstate: current_term: 1 leader_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:43.191417 29413 tablet_bootstrap.cc:492] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0: No bootstrap required, opened a new log
I20240503 08:07:43.191460 29413 ts_tablet_manager.cc:1367] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:43.191560 29413 raft_consensus.cc:348] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } }
I20240503 08:07:43.191620 29413 raft_consensus.cc:374] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:43.191643 29413 raft_consensus.cc:729] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a79d7e5f8254c71b41b9d6d89e60fd0, State: Initialized, Role: FOLLOWER
I20240503 08:07:43.191684 29413 consensus_queue.cc:260] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } }
I20240503 08:07:43.191717 29413 raft_consensus.cc:388] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:43.191733 29413 raft_consensus.cc:482] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:43.191749 29413 raft_consensus.cc:3037] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:43.192373 29413 raft_consensus.cc:504] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } }
I20240503 08:07:43.192433 29413 leader_election.cc:302] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a79d7e5f8254c71b41b9d6d89e60fd0; no voters:
I20240503 08:07:43.192502 29413 leader_election.cc:288] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:43.192548 29427 raft_consensus.cc:2781] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:43.192620 29427 raft_consensus.cc:686] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 LEADER]: Becoming Leader. State: Replica: 8a79d7e5f8254c71b41b9d6d89e60fd0, State: Running, Role: LEADER
I20240503 08:07:43.192672 29427 consensus_queue.cc:237] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } }
I20240503 08:07:43.192565 29413 ts_tablet_manager.cc:1398] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:43.193468 29055 catalog_manager.cc:5484] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a79d7e5f8254c71b41b9d6d89e60fd0 (127.28.55.65). New cstate: current_term: 1 leader_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a79d7e5f8254c71b41b9d6d89e60fd0" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 35631 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:43.194626 29420 raft_consensus.cc:2781] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:43.194844 29420 raft_consensus.cc:686] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 1 LEADER]: Becoming Leader. State: Replica: 54e1bcdd99bc413f9212fe3606983b88, State: Running, Role: LEADER
I20240503 08:07:43.194900 29420 consensus_queue.cc:237] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e1bcdd99bc413f9212fe3606983b88" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41811 } }
I20240503 08:07:43.195258 29055 catalog_manager.cc:5484] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54e1bcdd99bc413f9212fe3606983b88 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "54e1bcdd99bc413f9212fe3606983b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e1bcdd99bc413f9212fe3606983b88" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41811 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:43.198563 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:07:43.201248 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:43.201426 28893 tablet_replica.cc:330] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0: stopping tablet replica
I20240503 08:07:43.201520 28893 raft_consensus.cc:2229] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:43.201568 28893 raft_consensus.cc:2258] T bf70b97dd01f4fe09cf9b43383db533a P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:43.201894 28893 tablet_replica.cc:330] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0: stopping tablet replica
I20240503 08:07:43.201936 28893 raft_consensus.cc:2229] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:43.201963 28893 raft_consensus.cc:2258] T 413dc32531c849599cfe144d8fa66368 P 8a79d7e5f8254c71b41b9d6d89e60fd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:43.214293 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:07:43.215540 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:07:43.218073 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:43.218212 28893 tablet_replica.cc:330] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e: stopping tablet replica
I20240503 08:07:43.218277 28893 raft_consensus.cc:2229] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:43.218328 28893 raft_consensus.cc:2258] T 590dc86ae5794cb78377fcafeaff386f P 5df5736f58de4bfc9ecaf8ec39e2be2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:43.230476 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:07:43.231433 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:07:43.234644 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:43.234788 28893 tablet_replica.cc:330] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88: stopping tablet replica
I20240503 08:07:43.234879 28893 raft_consensus.cc:2229] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:43.234936 28893 raft_consensus.cc:2258] T 59853a8d2bf94df18653db72858933df P 54e1bcdd99bc413f9212fe3606983b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:43.246927 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:07:43.247844 28893 master.cc:537] Master@127.28.55.126:38129 shutting down...
I20240503 08:07:43.250077 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:43.250162 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:43.250195 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 499b4241406248b198b19c38cc0436f9: stopping tablet replica
I20240503 08:07:43.262426 28893 master.cc:552] Master@127.28.55.126:38129 shutdown complete.
I20240503 08:07:43.263700 28893 master.cc:537] Master@127.28.55.125:33403 shutting down...
I20240503 08:07:43.265619 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:43.265700 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:43.265717 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P f09fa2338d734346ba0a50c8b5125131: stopping tablet replica
I20240503 08:07:43.267540 28893 master.cc:552] Master@127.28.55.125:33403 shutdown complete.
I20240503 08:07:43.268708 28893 master.cc:537] Master@127.28.55.124:42093 shutting down...
I20240503 08:07:43.270763 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:43.270946 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:43.270985 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P e329d9de4de546b9a45221503ed371da: stopping tablet replica
I20240503 08:07:43.282922 28893 master.cc:552] Master@127.28.55.124:42093 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2409 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 08:07:43.288367 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:33747
I20240503 08:07:43.288532 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:43.289690 29442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:43.289798 29439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:43.289842 29440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:43.289932 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:43.290023 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:43.290045 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:43.290055 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723663290056 us; error 0 us; skew 500 ppm
I20240503 08:07:43.290611 28893 webserver.cc:460] Webserver started at http://127.28.55.126:40109/ using document root <none> and password file <none>
I20240503 08:07:43.290688 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:43.290743 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:43.290787 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:43.291077 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "bd2ac4acf19b4239b1abc57e032afbce"
format_stamp: "Formatted at 2024-05-03 08:07:43 on dist-test-slave-mlgg"
I20240503 08:07:43.291934 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:43.292366 29447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.292511 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.292551 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "bd2ac4acf19b4239b1abc57e032afbce"
format_stamp: "Formatted at 2024-05-03 08:07:43 on dist-test-slave-mlgg"
I20240503 08:07:43.292599 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:43.309156 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:43.309361 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:43.311970 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:33747
I20240503 08:07:43.313972 29499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:33747 every 8 connection(s)
I20240503 08:07:43.314131 29500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:43.315233 29500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce: Bootstrap starting.
I20240503 08:07:43.315483 29500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:43.315974 29500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce: No bootstrap required, opened a new log
I20240503 08:07:43.316079 29500 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ac4acf19b4239b1abc57e032afbce" member_type: VOTER }
I20240503 08:07:43.316133 29500 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:43.316144 29500 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2ac4acf19b4239b1abc57e032afbce, State: Initialized, Role: FOLLOWER
I20240503 08:07:43.316183 29500 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ac4acf19b4239b1abc57e032afbce" member_type: VOTER }
I20240503 08:07:43.316231 29500 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:43.316251 29500 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:43.316268 29500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:43.316705 29500 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ac4acf19b4239b1abc57e032afbce" member_type: VOTER }
I20240503 08:07:43.316763 29500 leader_election.cc:302] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd2ac4acf19b4239b1abc57e032afbce; no voters:
I20240503 08:07:43.316846 29500 leader_election.cc:288] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:43.316912 29503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:43.316982 29500 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:43.317065 29503 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 1 LEADER]: Becoming Leader. State: Replica: bd2ac4acf19b4239b1abc57e032afbce, State: Running, Role: LEADER
I20240503 08:07:43.317121 29503 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ac4acf19b4239b1abc57e032afbce" member_type: VOTER }
I20240503 08:07:43.317322 29504 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd2ac4acf19b4239b1abc57e032afbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ac4acf19b4239b1abc57e032afbce" member_type: VOTER } }
I20240503 08:07:43.317332 29505 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd2ac4acf19b4239b1abc57e032afbce. Latest consensus state: current_term: 1 leader_uuid: "bd2ac4acf19b4239b1abc57e032afbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2ac4acf19b4239b1abc57e032afbce" member_type: VOTER } }
I20240503 08:07:43.317466 29504 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:43.317471 29505 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:43.317729 29510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:07:43.317862 29510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:07:43.318254 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:07:43.318379 29510 catalog_manager.cc:1347] Generated new cluster ID: 96bb2a391dc14afcb79b5cae06fb008c
I20240503 08:07:43.318413 29510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:07:43.325913 29510 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:07:43.326377 29510 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:07:43.331051 29510 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce: Generated new TSK 0
I20240503 08:07:43.331152 29510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:07:43.333987 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:43.335341 29522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:43.335423 29524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:43.335441 28893 server_base.cc:1034] running on GCE node
W20240503 08:07:43.335511 29521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:43.335641 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:43.335682 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:43.335704 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723663335705 us; error 0 us; skew 500 ppm
I20240503 08:07:43.336427 28893 webserver.cc:460] Webserver started at http://127.28.55.65:32795/ using document root <none> and password file <none>
I20240503 08:07:43.336531 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:43.336578 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:43.336629 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:43.336897 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "5f57b564e8b94282b39545623e5eb6bb"
format_stamp: "Formatted at 2024-05-03 08:07:43 on dist-test-slave-mlgg"
I20240503 08:07:43.337752 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:43.338169 29529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.338274 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.338315 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "5f57b564e8b94282b39545623e5eb6bb"
format_stamp: "Formatted at 2024-05-03 08:07:43 on dist-test-slave-mlgg"
I20240503 08:07:43.338363 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:43.370682 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:43.370922 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:43.371094 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:43.371296 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:43.371325 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.371353 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:43.371373 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.374089 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:33927
I20240503 08:07:43.374159 29592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:33927 every 8 connection(s)
I20240503 08:07:43.374485 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:43.375644 29597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:43.375674 29600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:43.375877 29598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:43.375882 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:43.376004 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:43.376037 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:43.376052 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723663376052 us; error 0 us; skew 500 ppm
I20240503 08:07:43.376749 28893 webserver.cc:460] Webserver started at http://127.28.55.66:44435/ using document root <none> and password file <none>
I20240503 08:07:43.376847 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:43.376890 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:43.376935 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:43.377231 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "00bf96464adb444d96ffc4bb50c73f6d"
format_stamp: "Formatted at 2024-05-03 08:07:43 on dist-test-slave-mlgg"
I20240503 08:07:43.378119 29593 heartbeater.cc:347] Connected to a master server at 127.28.55.126:33747
I20240503 08:07:43.378185 29593 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:43.378226 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:43.378304 29593 heartbeater.cc:511] Master 127.28.55.126:33747 requested a full tablet report, sending...
I20240503 08:07:43.378572 29464 ts_manager.cc:194] Registered new tserver with Master: 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927)
I20240503 08:07:43.378738 29605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.378871 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.378924 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "00bf96464adb444d96ffc4bb50c73f6d"
format_stamp: "Formatted at 2024-05-03 08:07:43 on dist-test-slave-mlgg"
I20240503 08:07:43.378973 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:43.379287 29464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43198
I20240503 08:07:43.384542 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:43.384752 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:43.384898 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:43.385082 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:43.385107 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.385133 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:43.385155 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.387799 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:45961
I20240503 08:07:43.387854 29668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:45961 every 8 connection(s)
I20240503 08:07:43.388154 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:43.389181 29672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:43.389269 29673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:43.389396 29675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:43.389945 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:43.390089 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:43.390125 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:43.390141 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723663390141 us; error 0 us; skew 500 ppm
I20240503 08:07:43.391023 28893 webserver.cc:460] Webserver started at http://127.28.55.67:36573/ using document root <none> and password file <none>
I20240503 08:07:43.391101 29669 heartbeater.cc:347] Connected to a master server at 127.28.55.126:33747
I20240503 08:07:43.391131 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:43.391176 29669 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:43.391192 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:43.391243 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:43.391292 29669 heartbeater.cc:511] Master 127.28.55.126:33747 requested a full tablet report, sending...
I20240503 08:07:43.391534 29464 ts_manager.cc:194] Registered new tserver with Master: 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:43.391557 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "600525e1cff9418f8d2a4fdd50c15386"
format_stamp: "Formatted at 2024-05-03 08:07:43 on dist-test-slave-mlgg"
I20240503 08:07:43.391875 29464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43206
I20240503 08:07:43.392417 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:43.392868 29680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.392983 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.393023 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "600525e1cff9418f8d2a4fdd50c15386"
format_stamp: "Formatted at 2024-05-03 08:07:43 on dist-test-slave-mlgg"
I20240503 08:07:43.393064 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:43.411934 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:43.412156 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:43.412355 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:43.412595 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:43.412624 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.412653 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:43.412673 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:43.415830 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:41045
I20240503 08:07:43.415933 29743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:41045 every 8 connection(s)
I20240503 08:07:43.419245 29744 heartbeater.cc:347] Connected to a master server at 127.28.55.126:33747
I20240503 08:07:43.419337 29744 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:43.419438 29744 heartbeater.cc:511] Master 127.28.55.126:33747 requested a full tablet report, sending...
I20240503 08:07:43.419682 29464 ts_manager.cc:194] Registered new tserver with Master: 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67:41045)
I20240503 08:07:43.420006 29464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43214
I20240503 08:07:43.420403 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004393252s
I20240503 08:07:44.379820 29593 heartbeater.cc:503] Master 127.28.55.126:33747 was elected leader, sending a full tablet report...
I20240503 08:07:44.392410 29669 heartbeater.cc:503] Master 127.28.55.126:33747 was elected leader, sending a full tablet report...
I20240503 08:07:44.420480 29744 heartbeater.cc:503] Master 127.28.55.126:33747 was elected leader, sending a full tablet report...
I20240503 08:07:44.445369 28893 test_util.cc:268] Using random seed: 921190324
I20240503 08:07:44.450739 29464 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43218:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:07:44.450950 29464 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:07:44.456521 29708 tablet_service.cc:1467] Processing CreateTablet for tablet 83708ada0c4b429c8ed9da2562fc786b (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 08:07:44.456714 29708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83708ada0c4b429c8ed9da2562fc786b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.457733 29632 tablet_service.cc:1467] Processing CreateTablet for tablet 34a6f6d0d1e74bcfafc6fd7e76ad5739 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 08:07:44.457811 29626 tablet_service.cc:1467] Processing CreateTablet for tablet fc9052f80b384baeaf6e520a6ddbdec9 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 08:07:44.457839 29631 tablet_service.cc:1467] Processing CreateTablet for tablet 31a73f4b644942bb8c7d23a66814300c (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 08:07:44.457883 29626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9052f80b384baeaf6e520a6ddbdec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.457733 29633 tablet_service.cc:1467] Processing CreateTablet for tablet 83708ada0c4b429c8ed9da2562fc786b (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 08:07:44.457993 29630 tablet_service.cc:1467] Processing CreateTablet for tablet 4e16030220b047db8a7c1473b9965a36 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 08:07:44.458022 29633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83708ada0c4b429c8ed9da2562fc786b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.458117 29629 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac0c68a03b14aabad66f7bba0489949 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 08:07:44.458187 29629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac0c68a03b14aabad66f7bba0489949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.458384 29707 tablet_service.cc:1467] Processing CreateTablet for tablet 34a6f6d0d1e74bcfafc6fd7e76ad5739 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 08:07:44.458456 29707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a6f6d0d1e74bcfafc6fd7e76ad5739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.458902 29706 tablet_service.cc:1467] Processing CreateTablet for tablet 31a73f4b644942bb8c7d23a66814300c (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 08:07:44.458984 29706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a73f4b644942bb8c7d23a66814300c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.459259 29632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a6f6d0d1e74bcfafc6fd7e76ad5739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.460024 29630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e16030220b047db8a7c1473b9965a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.461162 29631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a73f4b644942bb8c7d23a66814300c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.461202 29764 tablet_bootstrap.cc:492] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386: Bootstrap starting.
I20240503 08:07:44.461613 29764 tablet_bootstrap.cc:654] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.460033 29628 tablet_service.cc:1467] Processing CreateTablet for tablet a140e26f2aab47078e0be8c9913999e0 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 08:07:44.460361 29627 tablet_service.cc:1467] Processing CreateTablet for tablet ee83257e12c24cac92eccff908f665c9 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 08:07:44.460366 29704 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac0c68a03b14aabad66f7bba0489949 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 08:07:44.460518 29701 tablet_service.cc:1467] Processing CreateTablet for tablet fc9052f80b384baeaf6e520a6ddbdec9 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 08:07:44.460558 29702 tablet_service.cc:1467] Processing CreateTablet for tablet ee83257e12c24cac92eccff908f665c9 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 08:07:44.460453 29765 tablet_bootstrap.cc:492] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d: Bootstrap starting.
I20240503 08:07:44.460597 29705 tablet_service.cc:1467] Processing CreateTablet for tablet 4e16030220b047db8a7c1473b9965a36 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 08:07:44.460700 29557 tablet_service.cc:1467] Processing CreateTablet for tablet 83708ada0c4b429c8ed9da2562fc786b (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 08:07:44.462774 29557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83708ada0c4b429c8ed9da2562fc786b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.460728 29556 tablet_service.cc:1467] Processing CreateTablet for tablet 34a6f6d0d1e74bcfafc6fd7e76ad5739 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 08:07:44.463956 29764 tablet_bootstrap.cc:492] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386: No bootstrap required, opened a new log
I20240503 08:07:44.464025 29764 ts_tablet_manager.cc:1367] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:07:44.460798 29551 tablet_service.cc:1467] Processing CreateTablet for tablet ee83257e12c24cac92eccff908f665c9 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 08:07:44.460913 29553 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac0c68a03b14aabad66f7bba0489949 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 08:07:44.460947 29550 tablet_service.cc:1467] Processing CreateTablet for tablet fc9052f80b384baeaf6e520a6ddbdec9 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 08:07:44.460783 29552 tablet_service.cc:1467] Processing CreateTablet for tablet a140e26f2aab47078e0be8c9913999e0 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 08:07:44.462122 29704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac0c68a03b14aabad66f7bba0489949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.460880 29554 tablet_service.cc:1467] Processing CreateTablet for tablet 4e16030220b047db8a7c1473b9965a36 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 08:07:44.460479 29703 tablet_service.cc:1467] Processing CreateTablet for tablet a140e26f2aab47078e0be8c9913999e0 (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 08:07:44.464030 29556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a6f6d0d1e74bcfafc6fd7e76ad5739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.460845 29555 tablet_service.cc:1467] Processing CreateTablet for tablet 31a73f4b644942bb8c7d23a66814300c (DEFAULT_TABLE table=test-workload [id=c286dba470b54a069592fd1ed5dd6971]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 08:07:44.464359 29703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a140e26f2aab47078e0be8c9913999e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.464424 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a73f4b644942bb8c7d23a66814300c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.465451 29552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a140e26f2aab47078e0be8c9913999e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.465515 29705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e16030220b047db8a7c1473b9965a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.465560 29554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e16030220b047db8a7c1473b9965a36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.462049 29628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a140e26f2aab47078e0be8c9913999e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.466532 29553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac0c68a03b14aabad66f7bba0489949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.466547 29702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee83257e12c24cac92eccff908f665c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.466713 29627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee83257e12c24cac92eccff908f665c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.466828 29551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee83257e12c24cac92eccff908f665c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.467716 29701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9052f80b384baeaf6e520a6ddbdec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.467902 29550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9052f80b384baeaf6e520a6ddbdec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:44.468173 29765 tablet_bootstrap.cc:654] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.468806 29765 tablet_bootstrap.cc:492] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d: No bootstrap required, opened a new log
I20240503 08:07:44.468848 29765 ts_tablet_manager.cc:1367] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:07:44.468995 29765 raft_consensus.cc:348] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.469051 29765 raft_consensus.cc:374] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.469062 29765 raft_consensus.cc:729] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.469102 29765 consensus_queue.cc:260] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.469251 29765 ts_tablet_manager.cc:1398] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:44.469326 29765 tablet_bootstrap.cc:492] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d: Bootstrap starting.
I20240503 08:07:44.464211 29764 raft_consensus.cc:348] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.469664 29764 raft_consensus.cc:374] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.469686 29764 raft_consensus.cc:729] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.469765 29765 tablet_bootstrap.cc:654] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.469748 29764 consensus_queue.cc:260] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.469851 29767 tablet_bootstrap.cc:492] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb: Bootstrap starting.
I20240503 08:07:44.469930 29764 ts_tablet_manager.cc:1398] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386: Time spent starting tablet: real 0.006s user 0.000s sys 0.000s
I20240503 08:07:44.469986 29764 tablet_bootstrap.cc:492] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386: Bootstrap starting.
I20240503 08:07:44.470188 29767 tablet_bootstrap.cc:654] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.470337 29764 tablet_bootstrap.cc:654] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.470701 29767 tablet_bootstrap.cc:492] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb: No bootstrap required, opened a new log
I20240503 08:07:44.470744 29767 ts_tablet_manager.cc:1367] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:44.470887 29767 raft_consensus.cc:348] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.470950 29767 raft_consensus.cc:374] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.470970 29767 raft_consensus.cc:729] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.471041 29767 consensus_queue.cc:260] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.471153 29764 tablet_bootstrap.cc:492] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386: No bootstrap required, opened a new log
I20240503 08:07:44.471202 29764 ts_tablet_manager.cc:1367] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:44.471311 29765 tablet_bootstrap.cc:492] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d: No bootstrap required, opened a new log
I20240503 08:07:44.471350 29765 ts_tablet_manager.cc:1367] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:07:44.471356 29764 raft_consensus.cc:348] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.471493 29764 raft_consensus.cc:374] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.471589 29764 raft_consensus.cc:729] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.471498 29765 raft_consensus.cc:348] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.471647 29765 raft_consensus.cc:374] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.471630 29764 consensus_queue.cc:260] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.471670 29765 raft_consensus.cc:729] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.471724 29764 ts_tablet_manager.cc:1398] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.471731 29765 consensus_queue.cc:260] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.471787 29764 tablet_bootstrap.cc:492] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386: Bootstrap starting.
I20240503 08:07:44.471864 29765 ts_tablet_manager.cc:1398] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:44.471910 29765 tablet_bootstrap.cc:492] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d: Bootstrap starting.
I20240503 08:07:44.472260 29765 tablet_bootstrap.cc:654] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.472345 29764 tablet_bootstrap.cc:654] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.471199 29767 ts_tablet_manager.cc:1398] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:44.472700 29774 raft_consensus.cc:482] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:44.472755 29767 tablet_bootstrap.cc:492] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb: Bootstrap starting.
I20240503 08:07:44.472743 29774 raft_consensus.cc:504] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.472914 29774 leader_election.cc:288] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961), 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67:41045)
I20240503 08:07:44.473251 29764 tablet_bootstrap.cc:492] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386: No bootstrap required, opened a new log
I20240503 08:07:44.473299 29764 ts_tablet_manager.cc:1367] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:07:44.473428 29764 raft_consensus.cc:348] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.473488 29764 raft_consensus.cc:374] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.473506 29764 raft_consensus.cc:729] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.473542 29764 consensus_queue.cc:260] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.473618 29764 ts_tablet_manager.cc:1398] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.473664 29764 tablet_bootstrap.cc:492] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386: Bootstrap starting.
I20240503 08:07:44.474027 29764 tablet_bootstrap.cc:654] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.474359 29765 tablet_bootstrap.cc:492] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d: No bootstrap required, opened a new log
I20240503 08:07:44.474397 29765 ts_tablet_manager.cc:1367] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:07:44.474527 29765 raft_consensus.cc:348] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.474601 29765 raft_consensus.cc:374] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.474619 29765 raft_consensus.cc:729] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.474649 29767 tablet_bootstrap.cc:654] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.474675 29765 consensus_queue.cc:260] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.474884 29765 ts_tablet_manager.cc:1398] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.474951 29765 tablet_bootstrap.cc:492] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d: Bootstrap starting.
I20240503 08:07:44.475364 29765 tablet_bootstrap.cc:654] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.475857 29765 tablet_bootstrap.cc:492] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d: No bootstrap required, opened a new log
I20240503 08:07:44.475886 29765 ts_tablet_manager.cc:1367] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:44.475983 29765 raft_consensus.cc:348] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.476027 29765 raft_consensus.cc:374] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.476042 29765 raft_consensus.cc:729] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.476086 29765 consensus_queue.cc:260] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.476168 29765 ts_tablet_manager.cc:1398] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.476245 29765 tablet_bootstrap.cc:492] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d: Bootstrap starting.
I20240503 08:07:44.476588 29765 tablet_bootstrap.cc:654] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.476778 29718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83708ada0c4b429c8ed9da2562fc786b" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600525e1cff9418f8d2a4fdd50c15386" is_pre_election: true
I20240503 08:07:44.476794 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83708ada0c4b429c8ed9da2562fc786b" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d" is_pre_election: true
I20240503 08:07:44.476866 29718 raft_consensus.cc:2454] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 0.
I20240503 08:07:44.476876 29643 raft_consensus.cc:2454] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 0.
I20240503 08:07:44.477027 29530 leader_election.cc:302] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.477073 29764 tablet_bootstrap.cc:492] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386: No bootstrap required, opened a new log
I20240503 08:07:44.477114 29774 raft_consensus.cc:2781] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:44.477159 29774 raft_consensus.cc:482] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:44.477181 29774 raft_consensus.cc:3037] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.477380 29765 tablet_bootstrap.cc:492] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d: No bootstrap required, opened a new log
I20240503 08:07:44.477417 29765 ts_tablet_manager.cc:1367] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:44.477543 29765 raft_consensus.cc:348] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.477598 29765 raft_consensus.cc:374] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.477612 29765 raft_consensus.cc:729] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.477640 29765 consensus_queue.cc:260] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.477717 29765 ts_tablet_manager.cc:1398] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.477771 29765 tablet_bootstrap.cc:492] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d: Bootstrap starting.
I20240503 08:07:44.477871 29774 raft_consensus.cc:504] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.477977 29774 leader_election.cc:288] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 1 election: Requested vote from peers 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961), 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67:41045)
I20240503 08:07:44.477116 29764 ts_tablet_manager.cc:1367] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:07:44.478098 29765 tablet_bootstrap.cc:654] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.478180 29764 raft_consensus.cc:348] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.478233 29764 raft_consensus.cc:374] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.478257 29764 raft_consensus.cc:729] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.478302 29764 consensus_queue.cc:260] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.478384 29764 ts_tablet_manager.cc:1398] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.478432 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83708ada0c4b429c8ed9da2562fc786b" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
I20240503 08:07:44.478454 29764 tablet_bootstrap.cc:492] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386: Bootstrap starting.
I20240503 08:07:44.478474 29643 raft_consensus.cc:3037] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.478793 29764 tablet_bootstrap.cc:654] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.479038 29765 tablet_bootstrap.cc:492] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d: No bootstrap required, opened a new log
I20240503 08:07:44.479077 29765 ts_tablet_manager.cc:1367] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:44.479177 29767 tablet_bootstrap.cc:492] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb: No bootstrap required, opened a new log
I20240503 08:07:44.479198 29765 raft_consensus.cc:348] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.478364 29718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83708ada0c4b429c8ed9da2562fc786b" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
I20240503 08:07:44.479293 29765 raft_consensus.cc:374] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.479254 29767 ts_tablet_manager.cc:1367] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:07:44.479308 29718 raft_consensus.cc:3037] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.479450 29767 raft_consensus.cc:348] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.479496 29764 tablet_bootstrap.cc:492] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386: No bootstrap required, opened a new log
I20240503 08:07:44.479508 29767 raft_consensus.cc:374] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.479526 29767 raft_consensus.cc:729] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.479529 29764 ts_tablet_manager.cc:1367] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:44.479607 29767 consensus_queue.cc:260] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.479692 29767 ts_tablet_manager.cc:1398] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.479758 29767 tablet_bootstrap.cc:492] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb: Bootstrap starting.
I20240503 08:07:44.479991 29718 raft_consensus.cc:2454] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 1.
I20240503 08:07:44.480139 29767 tablet_bootstrap.cc:654] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.479943 29764 raft_consensus.cc:348] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.480223 29764 raft_consensus.cc:374] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.480245 29530 leader_election.cc:302] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.479179 29643 raft_consensus.cc:2454] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 1.
I20240503 08:07:44.479312 29765 raft_consensus.cc:729] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.480377 29765 consensus_queue.cc:260] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.480429 29774 raft_consensus.cc:2781] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:44.480471 29765 ts_tablet_manager.cc:1398] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:07:44.480525 29765 tablet_bootstrap.cc:492] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d: Bootstrap starting.
I20240503 08:07:44.480542 29774 raft_consensus.cc:686] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 1 LEADER]: Becoming Leader. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Running, Role: LEADER
I20240503 08:07:44.480590 29774 consensus_queue.cc:237] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.480834 29765 tablet_bootstrap.cc:654] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.481050 29463 catalog_manager.cc:5484] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65). New cstate: current_term: 1 leader_uuid: "5f57b564e8b94282b39545623e5eb6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:44.480254 29764 raft_consensus.cc:729] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.481381 29764 consensus_queue.cc:260] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.481410 29765 tablet_bootstrap.cc:492] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d: No bootstrap required, opened a new log
I20240503 08:07:44.481454 29765 ts_tablet_manager.cc:1367] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:44.481577 29765 raft_consensus.cc:348] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.481637 29765 raft_consensus.cc:374] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.481662 29765 raft_consensus.cc:729] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.481699 29765 consensus_queue.cc:260] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.481781 29765 ts_tablet_manager.cc:1398] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.481851 29765 tablet_bootstrap.cc:492] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d: Bootstrap starting.
I20240503 08:07:44.481861 29764 ts_tablet_manager.cc:1398] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:07:44.481981 29764 tablet_bootstrap.cc:492] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386: Bootstrap starting.
I20240503 08:07:44.482192 29765 tablet_bootstrap.cc:654] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.482429 29764 tablet_bootstrap.cc:654] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.482990 29765 tablet_bootstrap.cc:492] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d: No bootstrap required, opened a new log
I20240503 08:07:44.482975 29767 tablet_bootstrap.cc:492] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb: No bootstrap required, opened a new log
I20240503 08:07:44.483042 29764 tablet_bootstrap.cc:492] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386: No bootstrap required, opened a new log
I20240503 08:07:44.483055 29765 ts_tablet_manager.cc:1367] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:44.483080 29764 ts_tablet_manager.cc:1367] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:44.483067 29767 ts_tablet_manager.cc:1367] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20240503 08:07:44.483219 29764 raft_consensus.cc:348] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.483223 29767 raft_consensus.cc:348] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.483273 29764 raft_consensus.cc:374] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.483223 29765 raft_consensus.cc:348] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.483307 29767 raft_consensus.cc:374] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.483317 29764 raft_consensus.cc:729] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.483338 29765 raft_consensus.cc:374] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.483386 29764 consensus_queue.cc:260] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.483343 29767 raft_consensus.cc:729] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.483438 29765 raft_consensus.cc:729] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.483480 29764 ts_tablet_manager.cc:1398] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.483516 29767 consensus_queue.cc:260] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.483516 29765 consensus_queue.cc:260] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.483623 29765 ts_tablet_manager.cc:1398] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:44.483626 29767 ts_tablet_manager.cc:1398] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:07:44.483695 29767 tablet_bootstrap.cc:492] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb: Bootstrap starting.
I20240503 08:07:44.483866 29764 tablet_bootstrap.cc:492] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386: Bootstrap starting.
I20240503 08:07:44.484033 29767 tablet_bootstrap.cc:654] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.484238 29764 tablet_bootstrap.cc:654] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.484710 29767 tablet_bootstrap.cc:492] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb: No bootstrap required, opened a new log
I20240503 08:07:44.484759 29767 ts_tablet_manager.cc:1367] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:07:44.484896 29764 tablet_bootstrap.cc:492] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386: No bootstrap required, opened a new log
I20240503 08:07:44.484956 29764 ts_tablet_manager.cc:1367] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:44.484896 29767 raft_consensus.cc:348] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.484992 29767 raft_consensus.cc:374] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.485003 29767 raft_consensus.cc:729] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.485033 29767 consensus_queue.cc:260] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.485066 29764 raft_consensus.cc:348] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.485095 29767 ts_tablet_manager.cc:1398] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.485133 29764 raft_consensus.cc:374] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.485153 29764 raft_consensus.cc:729] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.485167 29767 tablet_bootstrap.cc:492] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb: Bootstrap starting.
I20240503 08:07:44.485186 29764 consensus_queue.cc:260] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.485268 29764 ts_tablet_manager.cc:1398] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.485312 29764 tablet_bootstrap.cc:492] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386: Bootstrap starting.
I20240503 08:07:44.485523 29767 tablet_bootstrap.cc:654] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.485944 29764 tablet_bootstrap.cc:654] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.486042 29767 tablet_bootstrap.cc:492] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb: No bootstrap required, opened a new log
I20240503 08:07:44.486083 29767 ts_tablet_manager.cc:1367] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:07:44.486199 29767 raft_consensus.cc:348] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.486243 29767 raft_consensus.cc:374] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.486254 29767 raft_consensus.cc:729] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.486285 29767 consensus_queue.cc:260] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.486356 29767 ts_tablet_manager.cc:1398] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.486414 29767 tablet_bootstrap.cc:492] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb: Bootstrap starting.
I20240503 08:07:44.486460 29764 tablet_bootstrap.cc:492] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386: No bootstrap required, opened a new log
I20240503 08:07:44.486498 29764 ts_tablet_manager.cc:1367] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:44.486670 29764 raft_consensus.cc:348] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.486721 29764 raft_consensus.cc:374] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.486739 29764 raft_consensus.cc:729] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.486783 29764 consensus_queue.cc:260] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.486905 29764 ts_tablet_manager.cc:1398] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.486908 29767 tablet_bootstrap.cc:654] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.487449 29767 tablet_bootstrap.cc:492] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb: No bootstrap required, opened a new log
I20240503 08:07:44.487495 29767 ts_tablet_manager.cc:1367] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:07:44.487596 29767 raft_consensus.cc:348] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.487636 29767 raft_consensus.cc:374] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.487648 29767 raft_consensus.cc:729] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.487677 29767 consensus_queue.cc:260] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.487749 29767 ts_tablet_manager.cc:1398] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.487841 29767 tablet_bootstrap.cc:492] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb: Bootstrap starting.
I20240503 08:07:44.488179 29767 tablet_bootstrap.cc:654] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.488742 29767 tablet_bootstrap.cc:492] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb: No bootstrap required, opened a new log
I20240503 08:07:44.488788 29767 ts_tablet_manager.cc:1367] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:07:44.489023 29767 raft_consensus.cc:348] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.489071 29767 raft_consensus.cc:374] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.489089 29767 raft_consensus.cc:729] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.489140 29767 consensus_queue.cc:260] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.489228 29767 ts_tablet_manager.cc:1398] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.489284 29767 tablet_bootstrap.cc:492] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb: Bootstrap starting.
I20240503 08:07:44.489595 29767 tablet_bootstrap.cc:654] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:44.490105 29767 tablet_bootstrap.cc:492] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb: No bootstrap required, opened a new log
I20240503 08:07:44.490147 29767 ts_tablet_manager.cc:1367] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:07:44.490257 29767 raft_consensus.cc:348] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.490298 29767 raft_consensus.cc:374] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:44.490309 29767 raft_consensus.cc:729] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Initialized, Role: FOLLOWER
I20240503 08:07:44.490336 29767 consensus_queue.cc:260] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.490406 29767 ts_tablet_manager.cc:1398] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.543234 29770 raft_consensus.cc:482] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:44.543350 29770 raft_consensus.cc:504] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.543601 29770 leader_election.cc:288] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.546356 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a6f6d0d1e74bcfafc6fd7e76ad5739" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d" is_pre_election: true
I20240503 08:07:44.546367 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a6f6d0d1e74bcfafc6fd7e76ad5739" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb" is_pre_election: true
I20240503 08:07:44.546504 29643 raft_consensus.cc:2454] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.546507 29567 raft_consensus.cc:2454] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.546697 29681 leader_election.cc:302] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.546844 29770 raft_consensus.cc:2781] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:44.546911 29770 raft_consensus.cc:482] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:44.546936 29770 raft_consensus.cc:3037] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.547616 29770 raft_consensus.cc:504] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.547787 29770 leader_election.cc:288] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Requested vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.547927 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a6f6d0d1e74bcfafc6fd7e76ad5739" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
I20240503 08:07:44.547968 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a6f6d0d1e74bcfafc6fd7e76ad5739" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
I20240503 08:07:44.547997 29643 raft_consensus.cc:3037] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.548008 29567 raft_consensus.cc:3037] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.548687 29643 raft_consensus.cc:2454] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.548835 29681 leader_election.cc:302] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.548921 29567 raft_consensus.cc:2454] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.548940 29770 raft_consensus.cc:2781] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:44.549094 29770 raft_consensus.cc:686] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Leader. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:44.549156 29770 consensus_queue.cc:237] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.549724 29463 catalog_manager.cc:5484] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "600525e1cff9418f8d2a4fdd50c15386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:44.594586 29774 raft_consensus.cc:482] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:44.594710 29774 raft_consensus.cc:504] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.595034 29774 leader_election.cc:288] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961), 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67:41045)
I20240503 08:07:44.595254 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee83257e12c24cac92eccff908f665c9" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d" is_pre_election: true
I20240503 08:07:44.595315 29643 raft_consensus.cc:2454] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 0.
I20240503 08:07:44.595285 29718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee83257e12c24cac92eccff908f665c9" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600525e1cff9418f8d2a4fdd50c15386" is_pre_election: true
I20240503 08:07:44.595381 29718 raft_consensus.cc:2454] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 0.
I20240503 08:07:44.595446 29530 leader_election.cc:302] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 5f57b564e8b94282b39545623e5eb6bb; no voters:
I20240503 08:07:44.595533 29774 raft_consensus.cc:2781] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:44.595572 29774 raft_consensus.cc:482] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:44.595592 29774 raft_consensus.cc:3037] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.596227 29774 raft_consensus.cc:504] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.596329 29774 leader_election.cc:288] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 1 election: Requested vote from peers 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961), 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67:41045)
I20240503 08:07:44.596494 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee83257e12c24cac92eccff908f665c9" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
I20240503 08:07:44.596542 29718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee83257e12c24cac92eccff908f665c9" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
I20240503 08:07:44.596565 29643 raft_consensus.cc:3037] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.596588 29718 raft_consensus.cc:3037] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.597101 29643 raft_consensus.cc:2454] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 1.
I20240503 08:07:44.597107 29718 raft_consensus.cc:2454] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 1.
I20240503 08:07:44.597280 29530 leader_election.cc:302] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.597362 29774 raft_consensus.cc:2781] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:44.597419 29774 raft_consensus.cc:686] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 1 LEADER]: Becoming Leader. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Running, Role: LEADER
I20240503 08:07:44.597470 29774 consensus_queue.cc:237] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.598111 29463 catalog_manager.cc:5484] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65). New cstate: current_term: 1 leader_uuid: "5f57b564e8b94282b39545623e5eb6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:44.617617 29770 raft_consensus.cc:482] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:44.617733 29770 raft_consensus.cc:504] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.617942 29770 leader_election.cc:288] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.618163 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e16030220b047db8a7c1473b9965a36" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d" is_pre_election: true
I20240503 08:07:44.618163 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e16030220b047db8a7c1473b9965a36" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb" is_pre_election: true
I20240503 08:07:44.618268 29643 raft_consensus.cc:2454] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.618270 29567 raft_consensus.cc:2454] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.618403 29681 leader_election.cc:302] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.618469 29770 raft_consensus.cc:2781] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:44.618500 29770 raft_consensus.cc:482] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:44.618525 29770 raft_consensus.cc:3037] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.619199 29770 raft_consensus.cc:504] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.619330 29770 leader_election.cc:288] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Requested vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.619470 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e16030220b047db8a7c1473b9965a36" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
I20240503 08:07:44.619498 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e16030220b047db8a7c1473b9965a36" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
I20240503 08:07:44.619536 29643 raft_consensus.cc:3037] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.619541 29567 raft_consensus.cc:3037] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.620155 29643 raft_consensus.cc:2454] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.620167 29567 raft_consensus.cc:2454] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.620337 29681 leader_election.cc:302] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.620453 29770 raft_consensus.cc:2781] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:44.620508 29770 raft_consensus.cc:686] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Leader. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:44.620556 29770 consensus_queue.cc:237] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.621187 29463 catalog_manager.cc:5484] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "600525e1cff9418f8d2a4fdd50c15386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: HEALTHY } } }
W20240503 08:07:44.625516 29594 tablet.cc:2369] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:07:44.675896 29770 raft_consensus.cc:482] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:44.676020 29770 raft_consensus.cc:504] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.676204 29770 leader_election.cc:288] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.676436 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9052f80b384baeaf6e520a6ddbdec9" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb" is_pre_election: true
I20240503 08:07:44.676532 29567 raft_consensus.cc:2454] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.676440 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9052f80b384baeaf6e520a6ddbdec9" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d" is_pre_election: true
I20240503 08:07:44.676589 29643 raft_consensus.cc:2454] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.676751 29682 leader_election.cc:302] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.676865 29770 raft_consensus.cc:2781] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:44.676914 29770 raft_consensus.cc:482] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:44.676932 29770 raft_consensus.cc:3037] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.677570 29770 raft_consensus.cc:504] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.677690 29770 leader_election.cc:288] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Requested vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.677811 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9052f80b384baeaf6e520a6ddbdec9" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
I20240503 08:07:44.677856 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9052f80b384baeaf6e520a6ddbdec9" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
I20240503 08:07:44.677870 29567 raft_consensus.cc:3037] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.677937 29643 raft_consensus.cc:3037] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.678488 29643 raft_consensus.cc:2454] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.678488 29567 raft_consensus.cc:2454] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.678673 29682 leader_election.cc:302] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.678795 29770 raft_consensus.cc:2781] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:44.678870 29770 raft_consensus.cc:686] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Leader. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:44.678927 29770 consensus_queue.cc:237] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.679555 29463 catalog_manager.cc:5484] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "600525e1cff9418f8d2a4fdd50c15386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:44.689023 29770 raft_consensus.cc:482] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:44.689129 29770 raft_consensus.cc:504] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.689288 29770 leader_election.cc:288] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.689518 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a140e26f2aab47078e0be8c9913999e0" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d" is_pre_election: true
I20240503 08:07:44.689518 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a140e26f2aab47078e0be8c9913999e0" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb" is_pre_election: true
I20240503 08:07:44.689617 29643 raft_consensus.cc:2454] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.689627 29567 raft_consensus.cc:2454] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.689771 29682 leader_election.cc:302] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.689919 29770 raft_consensus.cc:2781] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:44.689952 29770 raft_consensus.cc:482] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:44.689970 29770 raft_consensus.cc:3037] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.690562 29770 raft_consensus.cc:504] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.690702 29770 leader_election.cc:288] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Requested vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.690877 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a140e26f2aab47078e0be8c9913999e0" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
I20240503 08:07:44.690878 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a140e26f2aab47078e0be8c9913999e0" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
I20240503 08:07:44.690956 29643 raft_consensus.cc:3037] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.690980 29567 raft_consensus.cc:3037] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.691462 29643 raft_consensus.cc:2454] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.691468 29567 raft_consensus.cc:2454] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.691641 29681 leader_election.cc:302] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.691751 29770 raft_consensus.cc:2781] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:44.691809 29770 raft_consensus.cc:686] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Leader. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:44.691864 29770 consensus_queue.cc:237] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.692468 29463 catalog_manager.cc:5484] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "600525e1cff9418f8d2a4fdd50c15386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:44.713258 29770 raft_consensus.cc:482] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:44.713387 29770 raft_consensus.cc:504] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.713588 29770 leader_election.cc:288] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.713846 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac0c68a03b14aabad66f7bba0489949" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d" is_pre_election: true
I20240503 08:07:44.713846 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac0c68a03b14aabad66f7bba0489949" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb" is_pre_election: true
I20240503 08:07:44.713984 29643 raft_consensus.cc:2454] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.713987 29567 raft_consensus.cc:2454] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.714143 29682 leader_election.cc:302] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.714293 29770 raft_consensus.cc:2781] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:44.714323 29770 raft_consensus.cc:482] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:44.714341 29770 raft_consensus.cc:3037] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.715049 29770 raft_consensus.cc:504] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.715193 29770 leader_election.cc:288] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Requested vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.715373 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac0c68a03b14aabad66f7bba0489949" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
I20240503 08:07:44.715373 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac0c68a03b14aabad66f7bba0489949" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
I20240503 08:07:44.715484 29567 raft_consensus.cc:3037] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.715488 29643 raft_consensus.cc:3037] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.716135 29643 raft_consensus.cc:2454] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.716135 29567 raft_consensus.cc:2454] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.716336 29682 leader_election.cc:302] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.716426 29770 raft_consensus.cc:2781] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:44.716471 29770 raft_consensus.cc:686] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Leader. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:44.716526 29770 consensus_queue.cc:237] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.717128 29463 catalog_manager.cc:5484] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "600525e1cff9418f8d2a4fdd50c15386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:44.824405 29770 raft_consensus.cc:482] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:44.824525 29770 raft_consensus.cc:504] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.824709 29770 leader_election.cc:288] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.824955 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a73f4b644942bb8c7d23a66814300c" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb" is_pre_election: true
I20240503 08:07:44.825049 29567 raft_consensus.cc:2454] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.824959 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a73f4b644942bb8c7d23a66814300c" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d" is_pre_election: true
I20240503 08:07:44.825095 29643 raft_consensus.cc:2454] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 0.
I20240503 08:07:44.825227 29682 leader_election.cc:302] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.825327 29770 raft_consensus.cc:2781] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:44.825368 29770 raft_consensus.cc:482] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:44.825387 29770 raft_consensus.cc:3037] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.825992 29770 raft_consensus.cc:504] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.826135 29770 leader_election.cc:288] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Requested vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961)
I20240503 08:07:44.826282 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a73f4b644942bb8c7d23a66814300c" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
I20240503 08:07:44.826282 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a73f4b644942bb8c7d23a66814300c" candidate_uuid: "600525e1cff9418f8d2a4fdd50c15386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
I20240503 08:07:44.826337 29567 raft_consensus.cc:3037] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.826359 29643 raft_consensus.cc:3037] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:44.826829 29567 raft_consensus.cc:2454] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.826925 29643 raft_consensus.cc:2454] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600525e1cff9418f8d2a4fdd50c15386 in term 1.
I20240503 08:07:44.827004 29682 leader_election.cc:302] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f57b564e8b94282b39545623e5eb6bb, 600525e1cff9418f8d2a4fdd50c15386; no voters:
I20240503 08:07:44.827108 29770 raft_consensus.cc:2781] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:44.827162 29770 raft_consensus.cc:686] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Leader. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:44.827212 29770 consensus_queue.cc:237] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:44.827828 29463 catalog_manager.cc:5484] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "600525e1cff9418f8d2a4fdd50c15386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:44.868211 29774 consensus_queue.cc:1035] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:44.869783 29774 consensus_queue.cc:1035] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:44.895990 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:44.897224 29816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:44.897289 29813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:44.897390 28893 server_base.cc:1034] running on GCE node
W20240503 08:07:44.897415 29814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:44.897629 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:44.897663 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:44.897675 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723664897675 us; error 0 us; skew 500 ppm
I20240503 08:07:44.898429 28893 webserver.cc:460] Webserver started at http://127.28.55.68:33081/ using document root <none> and password file <none>
I20240503 08:07:44.898509 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:44.898557 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:44.898599 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:44.898882 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-3-root/instance:
uuid: "53a1abd77cc0449d97b234d2060b80c3"
format_stamp: "Formatted at 2024-05-03 08:07:44 on dist-test-slave-mlgg"
I20240503 08:07:44.899680 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:44.900075 29821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.900204 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.900250 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-3-root
uuid: "53a1abd77cc0449d97b234d2060b80c3"
format_stamp: "Formatted at 2024-05-03 08:07:44 on dist-test-slave-mlgg"
I20240503 08:07:44.900290 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714723660874102-28893-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:44.904757 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:44.904973 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:44.905104 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:44.905288 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:44.905315 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.905334 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:44.905357 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:44.908849 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.68:33623
I20240503 08:07:44.908906 29884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.68:33623 every 8 connection(s)
I20240503 08:07:44.911841 29885 heartbeater.cc:347] Connected to a master server at 127.28.55.126:33747
I20240503 08:07:44.911908 29885 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:44.912017 29885 heartbeater.cc:511] Master 127.28.55.126:33747 requested a full tablet report, sending...
I20240503 08:07:44.912235 29464 ts_manager.cc:194] Registered new tserver with Master: 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68:33623)
I20240503 08:07:44.912580 29464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43226
I20240503 08:07:44.996613 29770 consensus_queue.cc:1035] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:44.998150 29800 consensus_queue.cc:1035] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.006886 29774 consensus_queue.cc:1035] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.008337 29774 consensus_queue.cc:1035] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.127180 29800 consensus_queue.cc:1035] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.128574 29770 consensus_queue.cc:1035] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.133255 29770 consensus_queue.cc:1035] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.134471 29800 consensus_queue.cc:1035] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.156591 29800 consensus_queue.cc:1035] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.158124 29898 consensus_queue.cc:1035] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.232368 29800 consensus_queue.cc:1035] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.232544 29898 consensus_queue.cc:1035] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.318324 29518 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:07:45.321885 29718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31a73f4b644942bb8c7d23a66814300c"
dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
mode: GRACEFUL
new_leader_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
from {username='slave'} at 127.0.0.1:53174
I20240503 08:07:45.322021 29718 raft_consensus.cc:595] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Received request to transfer leadership to TS 00bf96464adb444d96ffc4bb50c73f6d
I20240503 08:07:45.322458 29718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ac0c68a03b14aabad66f7bba0489949"
dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
mode: GRACEFUL
new_leader_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
from {username='slave'} at 127.0.0.1:53174
I20240503 08:07:45.322527 29718 raft_consensus.cc:595] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Received request to transfer leadership to TS 00bf96464adb444d96ffc4bb50c73f6d
I20240503 08:07:45.322793 29770 consensus_queue.cc:1035] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.322937 29718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e16030220b047db8a7c1473b9965a36"
dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
mode: GRACEFUL
new_leader_uuid: "5f57b564e8b94282b39545623e5eb6bb"
from {username='slave'} at 127.0.0.1:53174
I20240503 08:07:45.323010 29718 raft_consensus.cc:595] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Received request to transfer leadership to TS 5f57b564e8b94282b39545623e5eb6bb
I20240503 08:07:45.323331 29718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc9052f80b384baeaf6e520a6ddbdec9"
dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
mode: GRACEFUL
new_leader_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
from {username='slave'} at 127.0.0.1:53174
I20240503 08:07:45.323406 29718 raft_consensus.cc:595] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Received request to transfer leadership to TS 00bf96464adb444d96ffc4bb50c73f6d
I20240503 08:07:45.323545 29518 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240503 08:07:45.323619 29518 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:07:45.324188 29770 consensus_queue.cc:1035] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:45.324970 29800 raft_consensus.cc:979] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386: : Instructing follower 00bf96464adb444d96ffc4bb50c73f6d to start an election
I20240503 08:07:45.325027 29800 raft_consensus.cc:1067] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Signalling peer 00bf96464adb444d96ffc4bb50c73f6d to start an election
I20240503 08:07:45.325192 29643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31a73f4b644942bb8c7d23a66814300c"
dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
from {username='slave'} at 127.0.0.1:47708
I20240503 08:07:45.325245 29643 raft_consensus.cc:482] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:07:45.325273 29643 raft_consensus.cc:3037] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.325838 29643 raft_consensus.cc:504] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.325999 29643 leader_election.cc:288] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [CANDIDATE]: Term 2 election: Requested vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67:41045)
I20240503 08:07:45.329602 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a73f4b644942bb8c7d23a66814300c" candidate_uuid: "00bf96464adb444d96ffc4bb50c73f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
I20240503 08:07:45.329671 29718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a73f4b644942bb8c7d23a66814300c" candidate_uuid: "00bf96464adb444d96ffc4bb50c73f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
I20240503 08:07:45.329697 29567 raft_consensus.cc:3037] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.329792 29718 raft_consensus.cc:3032] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:07:45.329825 29718 raft_consensus.cc:729] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:45.329880 29718 consensus_queue.cc:260] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.329960 29718 raft_consensus.cc:3037] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.330406 29567 raft_consensus.cc:2454] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf96464adb444d96ffc4bb50c73f6d in term 2.
I20240503 08:07:45.330468 29718 raft_consensus.cc:2454] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf96464adb444d96ffc4bb50c73f6d in term 2.
I20240503 08:07:45.330616 29607 leader_election.cc:302] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 5f57b564e8b94282b39545623e5eb6bb; no voters:
I20240503 08:07:45.330731 29769 raft_consensus.cc:2781] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:07:45.330876 29769 raft_consensus.cc:686] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 2 LEADER]: Becoming Leader. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Running, Role: LEADER
I20240503 08:07:45.330929 29769 consensus_queue.cc:237] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.331444 29462 catalog_manager.cc:5484] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d reported cstate change: term changed from 1 to 2, leader changed from 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67) to 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66). New cstate: current_term: 2 leader_uuid: "00bf96464adb444d96ffc4bb50c73f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:45.523433 29770 raft_consensus.cc:979] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386: : Instructing follower 00bf96464adb444d96ffc4bb50c73f6d to start an election
I20240503 08:07:45.523569 29770 raft_consensus.cc:1067] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Signalling peer 00bf96464adb444d96ffc4bb50c73f6d to start an election
I20240503 08:07:45.523768 29643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ac0c68a03b14aabad66f7bba0489949"
dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
from {username='slave'} at 127.0.0.1:47708
I20240503 08:07:45.523866 29643 raft_consensus.cc:482] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:07:45.523890 29643 raft_consensus.cc:3037] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.524574 29643 raft_consensus.cc:504] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.524725 29643 leader_election.cc:288] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [CANDIDATE]: Term 2 election: Requested vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67:41045)
I20240503 08:07:45.524905 29718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac0c68a03b14aabad66f7bba0489949" candidate_uuid: "00bf96464adb444d96ffc4bb50c73f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
I20240503 08:07:45.524972 29718 raft_consensus.cc:3032] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:07:45.524989 29718 raft_consensus.cc:729] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:45.524912 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac0c68a03b14aabad66f7bba0489949" candidate_uuid: "00bf96464adb444d96ffc4bb50c73f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
I20240503 08:07:45.525019 29567 raft_consensus.cc:3037] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.525031 29718 consensus_queue.cc:260] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.525125 29718 raft_consensus.cc:3037] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.525648 29718 raft_consensus.cc:2454] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf96464adb444d96ffc4bb50c73f6d in term 2.
I20240503 08:07:45.525646 29567 raft_consensus.cc:2454] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf96464adb444d96ffc4bb50c73f6d in term 2.
I20240503 08:07:45.525848 29607 leader_election.cc:302] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 5f57b564e8b94282b39545623e5eb6bb; no voters:
I20240503 08:07:45.525967 29769 raft_consensus.cc:2781] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:07:45.526026 29769 raft_consensus.cc:686] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 LEADER]: Becoming Leader. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Running, Role: LEADER
I20240503 08:07:45.526084 29769 consensus_queue.cc:237] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.526578 29462 catalog_manager.cc:5484] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d reported cstate change: term changed from 1 to 2, leader changed from 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67) to 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66). New cstate: current_term: 2 leader_uuid: "00bf96464adb444d96ffc4bb50c73f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:45.549809 29770 raft_consensus.cc:979] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386: : Instructing follower 5f57b564e8b94282b39545623e5eb6bb to start an election
I20240503 08:07:45.549918 29770 raft_consensus.cc:1067] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Signalling peer 5f57b564e8b94282b39545623e5eb6bb to start an election
I20240503 08:07:45.550123 29567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e16030220b047db8a7c1473b9965a36"
dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
from {username='slave'} at 127.0.0.1:41962
I20240503 08:07:45.550220 29567 raft_consensus.cc:482] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:07:45.550241 29567 raft_consensus.cc:3037] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.550876 29567 raft_consensus.cc:504] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.551023 29567 leader_election.cc:288] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 2 election: Requested vote from peers 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66:45961), 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67:41045)
I20240503 08:07:45.551234 29718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e16030220b047db8a7c1473b9965a36" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
I20240503 08:07:45.551240 29643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e16030220b047db8a7c1473b9965a36" candidate_uuid: "5f57b564e8b94282b39545623e5eb6bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
I20240503 08:07:45.551311 29718 raft_consensus.cc:3032] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:07:45.551329 29643 raft_consensus.cc:3037] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.551332 29718 raft_consensus.cc:729] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:45.551429 29718 consensus_queue.cc:260] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.551549 29718 raft_consensus.cc:3037] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.551967 29643 raft_consensus.cc:2454] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 2.
I20240503 08:07:45.552049 29718 raft_consensus.cc:2454] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f57b564e8b94282b39545623e5eb6bb in term 2.
I20240503 08:07:45.552161 29530 leader_election.cc:302] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 5f57b564e8b94282b39545623e5eb6bb; no voters:
I20240503 08:07:45.552263 29774 raft_consensus.cc:2781] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:07:45.552316 29774 raft_consensus.cc:686] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 2 LEADER]: Becoming Leader. State: Replica: 5f57b564e8b94282b39545623e5eb6bb, State: Running, Role: LEADER
I20240503 08:07:45.552356 29774 consensus_queue.cc:237] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.552850 29462 catalog_manager.cc:5484] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb reported cstate change: term changed from 1 to 2, leader changed from 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67) to 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65). New cstate: current_term: 2 leader_uuid: "5f57b564e8b94282b39545623e5eb6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:45.713593 29567 raft_consensus.cc:1261] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Refusing update from remote peer 00bf96464adb444d96ffc4bb50c73f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:45.713920 29769 consensus_queue.cc:1035] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:45.715056 29718 raft_consensus.cc:1261] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Refusing update from remote peer 00bf96464adb444d96ffc4bb50c73f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:45.715263 29913 consensus_queue.cc:1035] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:45.827436 29800 raft_consensus.cc:979] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386: : Instructing follower 00bf96464adb444d96ffc4bb50c73f6d to start an election
I20240503 08:07:45.827528 29800 raft_consensus.cc:1067] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Signalling peer 00bf96464adb444d96ffc4bb50c73f6d to start an election
I20240503 08:07:45.827736 29643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc9052f80b384baeaf6e520a6ddbdec9"
dest_uuid: "00bf96464adb444d96ffc4bb50c73f6d"
from {username='slave'} at 127.0.0.1:47708
I20240503 08:07:45.827833 29643 raft_consensus.cc:482] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:07:45.827857 29643 raft_consensus.cc:3037] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.828461 29643 raft_consensus.cc:504] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.828598 29643 leader_election.cc:288] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [CANDIDATE]: Term 2 election: Requested vote from peers 5f57b564e8b94282b39545623e5eb6bb (127.28.55.65:33927), 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67:41045)
I20240503 08:07:45.828749 29567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9052f80b384baeaf6e520a6ddbdec9" candidate_uuid: "00bf96464adb444d96ffc4bb50c73f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5f57b564e8b94282b39545623e5eb6bb"
I20240503 08:07:45.828758 29718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9052f80b384baeaf6e520a6ddbdec9" candidate_uuid: "00bf96464adb444d96ffc4bb50c73f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "600525e1cff9418f8d2a4fdd50c15386"
I20240503 08:07:45.828852 29567 raft_consensus.cc:3037] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.828866 29718 raft_consensus.cc:3032] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:07:45.828891 29718 raft_consensus.cc:729] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 600525e1cff9418f8d2a4fdd50c15386, State: Running, Role: LEADER
I20240503 08:07:45.828958 29718 consensus_queue.cc:260] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.829034 29718 raft_consensus.cc:3037] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:45.829525 29567 raft_consensus.cc:2454] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf96464adb444d96ffc4bb50c73f6d in term 2.
I20240503 08:07:45.829679 29718 raft_consensus.cc:2454] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf96464adb444d96ffc4bb50c73f6d in term 2.
I20240503 08:07:45.829718 29607 leader_election.cc:302] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf96464adb444d96ffc4bb50c73f6d, 5f57b564e8b94282b39545623e5eb6bb; no voters:
I20240503 08:07:45.829792 29913 raft_consensus.cc:2781] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:07:45.829842 29913 raft_consensus.cc:686] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 LEADER]: Becoming Leader. State: Replica: 00bf96464adb444d96ffc4bb50c73f6d, State: Running, Role: LEADER
I20240503 08:07:45.829891 29913 consensus_queue.cc:237] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } }
I20240503 08:07:45.830349 29462 catalog_manager.cc:5484] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d reported cstate change: term changed from 1 to 2, leader changed from 600525e1cff9418f8d2a4fdd50c15386 (127.28.55.67) to 00bf96464adb444d96ffc4bb50c73f6d (127.28.55.66). New cstate: current_term: 2 leader_uuid: "00bf96464adb444d96ffc4bb50c73f6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:45.913085 29885 heartbeater.cc:503] Master 127.28.55.126:33747 was elected leader, sending a full tablet report...
I20240503 08:07:45.977550 29567 raft_consensus.cc:1261] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Refusing update from remote peer 00bf96464adb444d96ffc4bb50c73f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:45.977859 29913 consensus_queue.cc:1035] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:45.979079 29718 raft_consensus.cc:1261] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Refusing update from remote peer 00bf96464adb444d96ffc4bb50c73f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:45.979256 29913 consensus_queue.cc:1035] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:46.016814 29643 raft_consensus.cc:1261] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Refusing update from remote peer 5f57b564e8b94282b39545623e5eb6bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:46.017161 29774 consensus_queue.cc:1035] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:46.018090 29718 raft_consensus.cc:1261] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Refusing update from remote peer 5f57b564e8b94282b39545623e5eb6bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:46.018332 29786 consensus_queue.cc:1035] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:46.273852 29718 raft_consensus.cc:1261] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Refusing update from remote peer 00bf96464adb444d96ffc4bb50c73f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:46.274165 29909 consensus_queue.cc:1035] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:46.275393 29567 raft_consensus.cc:1261] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Refusing update from remote peer 00bf96464adb444d96ffc4bb50c73f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:46.275646 29909 consensus_queue.cc:1035] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:47.323817 29518 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:07:47.324056 29518 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:07:47.324097 29518 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:07:49.324265 29518 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:07:49.324513 29518 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:07:49.324554 29518 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:07:50.329021 29567 consensus_queue.cc:237] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } }
I20240503 08:07:50.329964 29718 raft_consensus.cc:1261] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Refusing update from remote peer 5f57b564e8b94282b39545623e5eb6bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:50.329958 29643 raft_consensus.cc:1261] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Refusing update from remote peer 5f57b564e8b94282b39545623e5eb6bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:50.330232 29922 consensus_queue.cc:1035] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:50.330384 29921 consensus_queue.cc:1035] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:50.331061 29922 raft_consensus.cc:2932] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.331393 29718 raft_consensus.cc:2932] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.332372 29643 raft_consensus.cc:2932] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.332406 29464 catalog_manager.cc:5484] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "5f57b564e8b94282b39545623e5eb6bb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
W20240503 08:07:50.333801 29530 consensus_peers.cc:485] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb -> Peer 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68:33623): Couldn't send request to peer 53a1abd77cc0449d97b234d2060b80c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee83257e12c24cac92eccff908f665c9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:07:50.335455 29718 consensus_queue.cc:237] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } }
I20240503 08:07:50.336105 29643 raft_consensus.cc:1261] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Refusing update from remote peer 600525e1cff9418f8d2a4fdd50c15386: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:50.336218 29567 raft_consensus.cc:1261] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Refusing update from remote peer 600525e1cff9418f8d2a4fdd50c15386: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:50.336321 29934 consensus_queue.cc:1035] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:50.336472 29933 consensus_queue.cc:1035] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:50.337074 29917 raft_consensus.cc:2932] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.338153 29567 raft_consensus.cc:2932] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.338227 29643 raft_consensus.cc:2932] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.338342 29718 consensus_queue.cc:237] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } attrs { replace: true } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } }
I20240503 08:07:50.338374 29463 catalog_manager.cc:5484] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "600525e1cff9418f8d2a4fdd50c15386" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:50.339512 29567 raft_consensus.cc:1261] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Refusing update from remote peer 600525e1cff9418f8d2a4fdd50c15386: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:07:50.339675 29681 consensus_peers.cc:485] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 -> Peer 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68:33623): Couldn't send request to peer 53a1abd77cc0449d97b234d2060b80c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34a6f6d0d1e74bcfafc6fd7e76ad5739. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:07:50.339687 29934 consensus_queue.cc:1035] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:07:50.339797 29681 consensus_peers.cc:485] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 -> Peer 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68:33623): Couldn't send request to peer 53a1abd77cc0449d97b234d2060b80c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a140e26f2aab47078e0be8c9913999e0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:07:50.339905 29643 raft_consensus.cc:1261] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Refusing update from remote peer 600525e1cff9418f8d2a4fdd50c15386: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:50.340071 29917 consensus_queue.cc:1035] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:50.340322 29934 raft_consensus.cc:2932] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } attrs { replace: true } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.340438 29567 raft_consensus.cc:2932] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } attrs { replace: true } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.341338 29462 catalog_manager.cc:5484] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "600525e1cff9418f8d2a4fdd50c15386" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:50.341567 29643 raft_consensus.cc:2932] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } attrs { replace: true } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.341588 29567 consensus_queue.cc:237] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } }
I20240503 08:07:50.342200 29718 raft_consensus.cc:1261] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Refusing update from remote peer 5f57b564e8b94282b39545623e5eb6bb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:07:50.342468 29922 consensus_queue.cc:1035] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:07:50.342554 29643 raft_consensus.cc:1261] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Refusing update from remote peer 5f57b564e8b94282b39545623e5eb6bb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240503 08:07:50.342566 29530 consensus_peers.cc:485] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb -> Peer 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68:33623): Couldn't send request to peer 53a1abd77cc0449d97b234d2060b80c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e16030220b047db8a7c1473b9965a36. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:07:50.342708 29922 consensus_queue.cc:1035] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:07:50.343211 29921 raft_consensus.cc:2932] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.343343 29718 raft_consensus.cc:2932] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.343602 29643 raft_consensus.cc:2932] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.344393 29463 catalog_manager.cc:5484] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 reported cstate change: config changed from index -1 to 3, NON_VOTER 53a1abd77cc0449d97b234d2060b80c3 (127.28.55.68) added. New cstate: current_term: 2 leader_uuid: "5f57b564e8b94282b39545623e5eb6bb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f57b564e8b94282b39545623e5eb6bb" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33927 } } peers { permanent_uuid: "00bf96464adb444d96ffc4bb50c73f6d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45961 } attrs { replace: true } } peers { permanent_uuid: "600525e1cff9418f8d2a4fdd50c15386" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 41045 } } peers { permanent_uuid: "53a1abd77cc0449d97b234d2060b80c3" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 33623 } attrs { promote: true } } }
I20240503 08:07:50.421906 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
W20240503 08:07:50.422644 29449 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20240503 08:07:50.423118 29512 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:07:50.423168 29512 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:07:50.423604 29512 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:07:50.423660 29512 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 08:07:50.425870 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:50.425997 28893 tablet_replica.cc:330] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb: stopping tablet replica
I20240503 08:07:50.426082 28893 raft_consensus.cc:2229] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.426131 28893 raft_consensus.cc:2258] T fc9052f80b384baeaf6e520a6ddbdec9 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.426316 28893 tablet_replica.cc:330] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb: stopping tablet replica
I20240503 08:07:50.426357 28893 raft_consensus.cc:2229] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.426410 28893 raft_consensus.cc:2258] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.426620 28893 tablet_replica.cc:330] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb: stopping tablet replica
I20240503 08:07:50.426669 28893 raft_consensus.cc:2229] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.426709 28893 raft_consensus.cc:2258] T 4ac0c68a03b14aabad66f7bba0489949 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.426859 28893 tablet_replica.cc:330] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb: stopping tablet replica
I20240503 08:07:50.426903 28893 raft_consensus.cc:2229] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.426985 28893 raft_consensus.cc:2258] T ee83257e12c24cac92eccff908f665c9 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.427167 28893 tablet_replica.cc:330] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb: stopping tablet replica
I20240503 08:07:50.427218 28893 raft_consensus.cc:2229] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.427300 28893 raft_consensus.cc:2258] T 83708ada0c4b429c8ed9da2562fc786b P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.427459 28893 tablet_replica.cc:330] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb: stopping tablet replica
I20240503 08:07:50.427502 28893 raft_consensus.cc:2229] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.427543 28893 raft_consensus.cc:2258] T a140e26f2aab47078e0be8c9913999e0 P 5f57b564e8b94282b39545623e5eb6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.427722 28893 tablet_replica.cc:330] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb: stopping tablet replica
I20240503 08:07:50.427764 28893 raft_consensus.cc:2229] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.427793 28893 raft_consensus.cc:2258] T 31a73f4b644942bb8c7d23a66814300c P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.427930 28893 tablet_replica.cc:330] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb: stopping tablet replica
I20240503 08:07:50.427963 28893 raft_consensus.cc:2229] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.428023 28893 raft_consensus.cc:2258] T 4e16030220b047db8a7c1473b9965a36 P 5f57b564e8b94282b39545623e5eb6bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.430505 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:07:50.431852 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:07:50.435858 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:50.435962 28893 tablet_replica.cc:330] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d: stopping tablet replica
I20240503 08:07:50.436022 28893 raft_consensus.cc:2229] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.436079 28893 raft_consensus.cc:2258] T a140e26f2aab47078e0be8c9913999e0 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.436327 28893 tablet_replica.cc:330] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d: stopping tablet replica
I20240503 08:07:50.436385 28893 raft_consensus.cc:2229] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.436429 28893 raft_consensus.cc:2258] T ee83257e12c24cac92eccff908f665c9 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.436623 28893 tablet_replica.cc:330] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d: stopping tablet replica
I20240503 08:07:50.436707 28893 raft_consensus.cc:2229] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.436749 28893 raft_consensus.cc:2258] T 83708ada0c4b429c8ed9da2562fc786b P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.436865 28893 tablet_replica.cc:330] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d: stopping tablet replica
I20240503 08:07:50.436914 28893 raft_consensus.cc:2229] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.436975 28893 raft_consensus.cc:2258] T 4ac0c68a03b14aabad66f7bba0489949 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.437083 28893 tablet_replica.cc:330] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d: stopping tablet replica
I20240503 08:07:50.437119 28893 raft_consensus.cc:2229] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.437191 28893 raft_consensus.cc:2258] T fc9052f80b384baeaf6e520a6ddbdec9 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.437311 28893 tablet_replica.cc:330] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d: stopping tablet replica
I20240503 08:07:50.437343 28893 raft_consensus.cc:2229] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.437373 28893 raft_consensus.cc:2258] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 00bf96464adb444d96ffc4bb50c73f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.437547 28893 tablet_replica.cc:330] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d: stopping tablet replica
I20240503 08:07:50.437588 28893 raft_consensus.cc:2229] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.437664 28893 raft_consensus.cc:2258] T 31a73f4b644942bb8c7d23a66814300c P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.437777 28893 tablet_replica.cc:330] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d: stopping tablet replica
I20240503 08:07:50.437809 28893 raft_consensus.cc:2229] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.437840 28893 raft_consensus.cc:2258] T 4e16030220b047db8a7c1473b9965a36 P 00bf96464adb444d96ffc4bb50c73f6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.439723 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:07:50.441303 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
W20240503 08:07:50.442584 29512 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:07:50.442653 29512 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:07:50.443689 29512 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:07:50.443733 29512 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 08:07:50.444160 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:50.444259 28893 tablet_replica.cc:330] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386: stopping tablet replica
I20240503 08:07:50.444320 28893 raft_consensus.cc:2229] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.444373 28893 raft_consensus.cc:2258] T ee83257e12c24cac92eccff908f665c9 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.444563 28893 tablet_replica.cc:330] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386: stopping tablet replica
I20240503 08:07:50.444602 28893 raft_consensus.cc:2229] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.444629 28893 raft_consensus.cc:2258] T 83708ada0c4b429c8ed9da2562fc786b P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.444716 28893 tablet_replica.cc:330] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386: stopping tablet replica
I20240503 08:07:50.444746 28893 raft_consensus.cc:2229] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.444762 28893 raft_consensus.cc:2258] T fc9052f80b384baeaf6e520a6ddbdec9 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.444833 28893 tablet_replica.cc:330] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386: stopping tablet replica
I20240503 08:07:50.444856 28893 raft_consensus.cc:2229] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.444904 28893 raft_consensus.cc:2258] T 34a6f6d0d1e74bcfafc6fd7e76ad5739 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.445075 28893 tablet_replica.cc:330] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386: stopping tablet replica
I20240503 08:07:50.445112 28893 raft_consensus.cc:2229] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.445138 28893 raft_consensus.cc:2258] T 4ac0c68a03b14aabad66f7bba0489949 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.445214 28893 tablet_replica.cc:330] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386: stopping tablet replica
I20240503 08:07:50.445247 28893 raft_consensus.cc:2229] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.445292 28893 raft_consensus.cc:2258] T a140e26f2aab47078e0be8c9913999e0 P 600525e1cff9418f8d2a4fdd50c15386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.445497 28893 tablet_replica.cc:330] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386: stopping tablet replica
I20240503 08:07:50.445544 28893 raft_consensus.cc:2229] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.445571 28893 raft_consensus.cc:2258] T 31a73f4b644942bb8c7d23a66814300c P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.445660 28893 tablet_replica.cc:330] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386: stopping tablet replica
I20240503 08:07:50.445686 28893 raft_consensus.cc:2229] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.445711 28893 raft_consensus.cc:2258] T 4e16030220b047db8a7c1473b9965a36 P 600525e1cff9418f8d2a4fdd50c15386 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.447706 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:07:50.448839 28893 tablet_server.cc:178] TabletServer@127.28.55.68:0 shutting down...
I20240503 08:07:50.451097 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:50.462477 28893 tablet_server.cc:194] TabletServer@127.28.55.68:0 shutdown complete.
I20240503 08:07:50.463376 28893 master.cc:537] Master@127.28.55.126:33747 shutting down...
I20240503 08:07:50.465529 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.465606 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.465623 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P bd2ac4acf19b4239b1abc57e032afbce: stopping tablet replica
I20240503 08:07:50.467363 28893 master.cc:552] Master@127.28.55.126:33747 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7186 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 08:07:50.475138 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:42917,127.28.55.125:44695,127.28.55.124:39287
I20240503 08:07:50.475302 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:50.476788 29945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.476912 28893 server_base.cc:1034] running on GCE node
W20240503 08:07:50.476938 29948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.477041 29946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.477173 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:50.477212 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.477228 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723670477228 us; error 0 us; skew 500 ppm
I20240503 08:07:50.477911 28893 webserver.cc:460] Webserver started at http://127.28.55.126:35127/ using document root <none> and password file <none>
I20240503 08:07:50.478009 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.478058 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.478104 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.478372 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "a47e8670945f441f910030ec750ba530"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.479224 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:50.479630 29953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.479754 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.479794 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "a47e8670945f441f910030ec750ba530"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.479854 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.496681 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.496910 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.499436 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:42917
I20240503 08:07:50.500015 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:07:50.503667 30005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:42917 every 8 connection(s)
I20240503 08:07:50.503803 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:07:50.504561 30009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.504539 30008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.504760 30011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.504873 30006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:50.504971 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:50.505044 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:50.505059 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.505069 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723670505070 us; error 0 us; skew 500 ppm
I20240503 08:07:50.505774 28893 webserver.cc:460] Webserver started at http://127.28.55.125:35953/ using document root <none> and password file <none>
I20240503 08:07:50.505863 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.505910 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.505962 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.506207 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root/instance:
uuid: "ab097550721e497e9e163b02cbb702c6"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.507086 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:50.507656 30018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.507804 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.507850 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root
uuid: "ab097550721e497e9e163b02cbb702c6"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.507901 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.508337 30006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:07:50.508934 30006 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.28.55.125:44695: Network error: Client connection negotiation failed: client connection to 127.28.55.125:44695: connect: Connection refused (error 111)
I20240503 08:07:50.512286 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.512491 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.514935 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.125:44695
I20240503 08:07:50.514976 30071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.125:44695 every 8 connection(s)
I20240503 08:07:50.515224 30072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:50.515233 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:07:50.516088 30072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:07:50.516381 30077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.516446 30075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.516389 30074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.516687 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:50.516800 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:50.516835 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.516851 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723670516852 us; error 0 us; skew 500 ppm
I20240503 08:07:50.517534 28893 webserver.cc:460] Webserver started at http://127.28.55.124:35119/ using document root <none> and password file <none>
I20240503 08:07:50.517604 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.517630 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.517657 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.517911 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root/instance:
uuid: "6cf50db4c4854323a91b6dc5974b3e04"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.519034 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:50.519587 30083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.519661 30072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:50.519698 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:50.519749 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root
uuid: "6cf50db4c4854323a91b6dc5974b3e04"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.519799 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.522470 30072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } has no permanent_uuid. Determining permanent_uuid...
W20240503 08:07:50.523097 30072 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.28.55.124:39287: Network error: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111)
I20240503 08:07:50.530313 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.530596 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.533361 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.124:39287
I20240503 08:07:50.533425 30137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.124:39287 every 8 connection(s)
I20240503 08:07:50.533579 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:07:50.533703 30138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:50.534076 30006 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } attempt: 1
I20240503 08:07:50.534883 30138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:50.537379 30006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:50.537591 30138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:50.540305 30138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } has no permanent_uuid. Determining permanent_uuid...
I20240503 08:07:50.541654 30006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530: Bootstrap starting.
I20240503 08:07:50.541988 30006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:50.542546 30006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530: No bootstrap required, opened a new log
I20240503 08:07:50.542680 30006 raft_consensus.cc:348] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:50.542758 30006 raft_consensus.cc:374] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:50.542778 30006 raft_consensus.cc:729] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47e8670945f441f910030ec750ba530, State: Initialized, Role: FOLLOWER
I20240503 08:07:50.542851 30006 consensus_queue.cc:260] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:50.543056 30006 sys_catalog.cc:564] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:50.543044 30144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.543170 30144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:50.543869 30138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04: Bootstrap starting.
I20240503 08:07:50.544170 30138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:50.544358 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240503 08:07:50.544471 30155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:07:50.544512 30155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:07:50.544719 30138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04: No bootstrap required, opened a new log
I20240503 08:07:50.544867 30138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:50.544924 30138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:50.544936 30138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf50db4c4854323a91b6dc5974b3e04, State: Initialized, Role: FOLLOWER
I20240503 08:07:50.544968 30138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:50.545145 30138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:50.545150 30157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.545248 30157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [sys.catalog]: This master's current role is: FOLLOWER
W20240503 08:07:50.546165 30168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:07:50.546200 30168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:07:50.578310 30072 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } attempt: 1
I20240503 08:07:50.582007 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6: Bootstrap starting.
I20240503 08:07:50.582408 30072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:50.583053 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6: No bootstrap required, opened a new log
I20240503 08:07:50.583252 30072 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:50.583321 30072 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:50.583339 30072 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab097550721e497e9e163b02cbb702c6, State: Initialized, Role: FOLLOWER
I20240503 08:07:50.583388 30072 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:50.583577 30072 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:50.583586 30170 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.583693 30170 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:50.584659 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240503 08:07:50.584745 30181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:07:50.584806 30181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:07:50.584932 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:50.586059 30183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.586094 30185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.586180 30182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.586292 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:50.586400 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:50.586426 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.586441 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723670586441 us; error 0 us; skew 500 ppm
I20240503 08:07:50.587293 28893 webserver.cc:460] Webserver started at http://127.28.55.65:41293/ using document root <none> and password file <none>
I20240503 08:07:50.587383 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.587426 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.587473 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.587730 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "f7a0e5fe62ac4150912713802b6a4a67"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.588512 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:50.588894 30190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.589018 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.589058 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "f7a0e5fe62ac4150912713802b6a4a67"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.589105 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.608312 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.608578 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.608757 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:50.608981 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:50.609011 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.609041 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:50.609063 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.612421 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:46177
I20240503 08:07:50.612530 30253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:46177 every 8 connection(s)
I20240503 08:07:50.612910 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:50.614825 30261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.615025 30264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.615278 30260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.615824 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:50.615979 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 08:07:50.616405 30255 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42917
W20240503 08:07:50.616405 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.616483 30255 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.616515 30256 heartbeater.cc:347] Connected to a master server at 127.28.55.125:44695
I20240503 08:07:50.616621 30256 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.616631 30254 heartbeater.cc:347] Connected to a master server at 127.28.55.124:39287
I20240503 08:07:50.616664 30254 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.616683 30255 heartbeater.cc:511] Master 127.28.55.126:42917 requested a full tablet report, sending...
I20240503 08:07:50.616729 30256 heartbeater.cc:511] Master 127.28.55.125:44695 requested a full tablet report, sending...
I20240503 08:07:50.616731 30254 heartbeater.cc:511] Master 127.28.55.124:39287 requested a full tablet report, sending...
I20240503 08:07:50.616562 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723670616562 us; error 0 us; skew 500 ppm
I20240503 08:07:50.617058 29970 ts_manager.cc:194] Registered new tserver with Master: f7a0e5fe62ac4150912713802b6a4a67 (127.28.55.65:46177)
I20240503 08:07:50.617058 30036 ts_manager.cc:194] Registered new tserver with Master: f7a0e5fe62ac4150912713802b6a4a67 (127.28.55.65:46177)
I20240503 08:07:50.617072 30102 ts_manager.cc:194] Registered new tserver with Master: f7a0e5fe62ac4150912713802b6a4a67 (127.28.55.65:46177)
I20240503 08:07:50.617694 28893 webserver.cc:460] Webserver started at http://127.28.55.66:38829/ using document root <none> and password file <none>
I20240503 08:07:50.617787 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.617834 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.617879 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.618146 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "e661dcbb6467447ca48e2fc307c8b786"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.619025 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:50.619482 30269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.619616 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.619663 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "e661dcbb6467447ca48e2fc307c8b786"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.619702 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.656832 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.657114 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.657320 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:50.657529 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:50.657562 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.657583 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:50.657606 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.661088 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:34035
I20240503 08:07:50.661141 30332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:34035 every 8 connection(s)
I20240503 08:07:50.661612 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:07:50.663414 28893 server_base.cc:1034] running on GCE node
W20240503 08:07:50.663472 30343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.663487 30340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.663560 30341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.663775 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:50.663810 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.663826 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723670663827 us; error 0 us; skew 500 ppm
I20240503 08:07:50.664769 28893 webserver.cc:460] Webserver started at http://127.28.55.67:45765/ using document root <none> and password file <none>
I20240503 08:07:50.664916 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.664979 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.665026 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.665176 30335 heartbeater.cc:347] Connected to a master server at 127.28.55.125:44695
I20240503 08:07:50.665239 30335 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.665346 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "a2b503e70bb942d2bdb86184be748082"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.665350 30335 heartbeater.cc:511] Master 127.28.55.125:44695 requested a full tablet report, sending...
I20240503 08:07:50.665740 30333 heartbeater.cc:347] Connected to a master server at 127.28.55.124:39287
I20240503 08:07:50.665741 30036 ts_manager.cc:194] Registered new tserver with Master: e661dcbb6467447ca48e2fc307c8b786 (127.28.55.66:34035)
I20240503 08:07:50.665809 30333 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.665904 30333 heartbeater.cc:511] Master 127.28.55.124:39287 requested a full tablet report, sending...
I20240503 08:07:50.665932 30334 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42917
I20240503 08:07:50.665980 30334 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.666088 30334 heartbeater.cc:511] Master 127.28.55.126:42917 requested a full tablet report, sending...
I20240503 08:07:50.666347 29970 ts_manager.cc:194] Registered new tserver with Master: e661dcbb6467447ca48e2fc307c8b786 (127.28.55.66:34035)
I20240503 08:07:50.666386 30102 ts_manager.cc:194] Registered new tserver with Master: e661dcbb6467447ca48e2fc307c8b786 (127.28.55.66:34035)
I20240503 08:07:50.666482 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:50.666963 30348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.667088 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.667132 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "a2b503e70bb942d2bdb86184be748082"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-mlgg"
I20240503 08:07:50.667171 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.675802 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.676043 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.676203 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:50.676416 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:50.676445 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.676470 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:50.676491 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.679659 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:44753
I20240503 08:07:50.679829 30411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:44753 every 8 connection(s)
I20240503 08:07:50.683319 30412 heartbeater.cc:347] Connected to a master server at 127.28.55.124:39287
I20240503 08:07:50.683395 30412 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.683487 30412 heartbeater.cc:511] Master 127.28.55.124:39287 requested a full tablet report, sending...
I20240503 08:07:50.683494 30414 heartbeater.cc:347] Connected to a master server at 127.28.55.125:44695
I20240503 08:07:50.683571 30414 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.683683 30414 heartbeater.cc:511] Master 127.28.55.125:44695 requested a full tablet report, sending...
I20240503 08:07:50.683773 30102 ts_manager.cc:194] Registered new tserver with Master: a2b503e70bb942d2bdb86184be748082 (127.28.55.67:44753)
I20240503 08:07:50.683970 30036 ts_manager.cc:194] Registered new tserver with Master: a2b503e70bb942d2bdb86184be748082 (127.28.55.67:44753)
I20240503 08:07:50.684047 30413 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42917
I20240503 08:07:50.684088 30413 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.684175 30413 heartbeater.cc:511] Master 127.28.55.126:42917 requested a full tablet report, sending...
I20240503 08:07:50.684396 29970 ts_manager.cc:194] Registered new tserver with Master: a2b503e70bb942d2bdb86184be748082 (127.28.55.67:44753)
I20240503 08:07:50.684944 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004963987s
I20240503 08:07:50.781136 30157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:50.781260 30157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:50.781452 30157 leader_election.cc:288] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47e8670945f441f910030ec750ba530 (127.28.55.126:42917), ab097550721e497e9e163b02cbb702c6 (127.28.55.125:44695)
I20240503 08:07:50.781749 30046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cf50db4c4854323a91b6dc5974b3e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab097550721e497e9e163b02cbb702c6" is_pre_election: true
I20240503 08:07:50.781749 29980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cf50db4c4854323a91b6dc5974b3e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e8670945f441f910030ec750ba530" is_pre_election: true
I20240503 08:07:50.781896 30046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf50db4c4854323a91b6dc5974b3e04 in term 0.
I20240503 08:07:50.781898 29980 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf50db4c4854323a91b6dc5974b3e04 in term 0.
I20240503 08:07:50.782097 30088 leader_election.cc:302] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf50db4c4854323a91b6dc5974b3e04, ab097550721e497e9e163b02cbb702c6; no voters:
I20240503 08:07:50.782238 30157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:50.782282 30157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:50.782295 30157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:50.782994 30157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:50.783176 30157 leader_election.cc:288] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [CANDIDATE]: Term 1 election: Requested vote from peers a47e8670945f441f910030ec750ba530 (127.28.55.126:42917), ab097550721e497e9e163b02cbb702c6 (127.28.55.125:44695)
I20240503 08:07:50.783357 29980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cf50db4c4854323a91b6dc5974b3e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47e8670945f441f910030ec750ba530"
I20240503 08:07:50.783336 30046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cf50db4c4854323a91b6dc5974b3e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab097550721e497e9e163b02cbb702c6"
I20240503 08:07:50.783442 29980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:50.783471 30046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:50.784181 30046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf50db4c4854323a91b6dc5974b3e04 in term 1.
I20240503 08:07:50.784183 29980 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf50db4c4854323a91b6dc5974b3e04 in term 1.
I20240503 08:07:50.784386 30085 leader_election.cc:302] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf50db4c4854323a91b6dc5974b3e04, a47e8670945f441f910030ec750ba530; no voters:
I20240503 08:07:50.784530 30157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:50.784665 30157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf50db4c4854323a91b6dc5974b3e04, State: Running, Role: LEADER
I20240503 08:07:50.784719 30157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:50.785073 30420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf50db4c4854323a91b6dc5974b3e04. Latest consensus state: current_term: 1 leader_uuid: "6cf50db4c4854323a91b6dc5974b3e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.785161 30420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:50.785356 30422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:07:50.785506 30422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:07:50.786118 29980 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 1 FOLLOWER]: Refusing update from remote peer 6cf50db4c4854323a91b6dc5974b3e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:50.786125 30046 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 1 FOLLOWER]: Refusing update from remote peer 6cf50db4c4854323a91b6dc5974b3e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:50.786342 30420 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:50.786435 30157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:50.786993 30144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf50db4c4854323a91b6dc5974b3e04. Latest consensus state: current_term: 1 leader_uuid: "6cf50db4c4854323a91b6dc5974b3e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.787075 30144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:50.787122 30170 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cf50db4c4854323a91b6dc5974b3e04. Latest consensus state: current_term: 1 leader_uuid: "6cf50db4c4854323a91b6dc5974b3e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.787163 30170 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:50.787568 30420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf50db4c4854323a91b6dc5974b3e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.787652 30420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:50.787715 30170 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf50db4c4854323a91b6dc5974b3e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.787770 30170 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:50.787814 30144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf50db4c4854323a91b6dc5974b3e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.787863 30144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:50.787608 30428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf50db4c4854323a91b6dc5974b3e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:50.788049 30422 catalog_manager.cc:1347] Generated new cluster ID: 757b1f287473459f8f66236bab2e4954
I20240503 08:07:50.788076 30428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:50.788084 30422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:07:50.798336 30422 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:07:50.798851 30422 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:07:50.815910 30422 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04: Generated new TSK 0
I20240503 08:07:50.816051 30422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:07:51.544823 30155 catalog_manager.cc:1259] Loaded cluster ID: 757b1f287473459f8f66236bab2e4954
I20240503 08:07:51.544935 30155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530: loading cluster ID for follower catalog manager: success
I20240503 08:07:51.545783 30155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530: acquiring CA information for follower catalog manager: success
I20240503 08:07:51.545953 30155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:07:51.585023 30181 catalog_manager.cc:1259] Loaded cluster ID: 757b1f287473459f8f66236bab2e4954
I20240503 08:07:51.585114 30181 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6: loading cluster ID for follower catalog manager: success
I20240503 08:07:51.585840 30181 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6: acquiring CA information for follower catalog manager: success
I20240503 08:07:51.585959 30181 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 08:07:51.618499 30102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40312
I20240503 08:07:51.667330 30102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40314
I20240503 08:07:51.684988 30102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40320
I20240503 08:07:52.546183 30167 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:07:52.619251 30254 heartbeater.cc:503] Master 127.28.55.124:39287 was elected leader, sending a full tablet report...
I20240503 08:07:52.667850 30333 heartbeater.cc:503] Master 127.28.55.124:39287 was elected leader, sending a full tablet report...
I20240503 08:07:52.685629 30412 heartbeater.cc:503] Master 127.28.55.124:39287 was elected leader, sending a full tablet report...
I20240503 08:07:52.937279 28893 test_util.cc:268] Using random seed: 929682236
I20240503 08:07:52.943135 30102 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40322:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 08:07:52.949784 30297 tablet_service.cc:1467] Processing CreateTablet for tablet a507c2f1baf043eca5d1055365ed1824 (DEFAULT_TABLE table=test-workload [id=85a640ba7ed646fc88eee63554e38a28]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 08:07:52.949962 30297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a507c2f1baf043eca5d1055365ed1824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:52.950006 30218 tablet_service.cc:1467] Processing CreateTablet for tablet 92c27a045bf34f30ad68b52fb21cab03 (DEFAULT_TABLE table=test-workload [id=85a640ba7ed646fc88eee63554e38a28]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 08:07:52.950124 30218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c27a045bf34f30ad68b52fb21cab03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:52.950461 30376 tablet_service.cc:1467] Processing CreateTablet for tablet d635b046e93b409f91929f465eee4919 (DEFAULT_TABLE table=test-workload [id=85a640ba7ed646fc88eee63554e38a28]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 08:07:52.950556 30376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d635b046e93b409f91929f465eee4919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:52.951520 30460 tablet_bootstrap.cc:492] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67: Bootstrap starting.
I20240503 08:07:52.951721 30462 tablet_bootstrap.cc:492] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082: Bootstrap starting.
I20240503 08:07:52.951897 30460 tablet_bootstrap.cc:654] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:52.952208 30462 tablet_bootstrap.cc:654] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:52.952246 30461 tablet_bootstrap.cc:492] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786: Bootstrap starting.
I20240503 08:07:52.952436 30460 tablet_bootstrap.cc:492] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67: No bootstrap required, opened a new log
I20240503 08:07:52.952472 30460 ts_tablet_manager.cc:1367] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:52.952616 30460 raft_consensus.cc:348] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0e5fe62ac4150912713802b6a4a67" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46177 } }
I20240503 08:07:52.952683 30460 raft_consensus.cc:374] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:52.952702 30460 raft_consensus.cc:729] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a0e5fe62ac4150912713802b6a4a67, State: Initialized, Role: FOLLOWER
I20240503 08:07:52.952724 30462 tablet_bootstrap.cc:492] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082: No bootstrap required, opened a new log
I20240503 08:07:52.952737 30460 consensus_queue.cc:260] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0e5fe62ac4150912713802b6a4a67" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46177 } }
I20240503 08:07:52.952772 30462 ts_tablet_manager.cc:1367] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:52.952778 30460 raft_consensus.cc:388] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:52.952837 30460 raft_consensus.cc:482] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:52.952857 30460 raft_consensus.cc:3037] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:52.952946 30462 raft_consensus.cc:348] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b503e70bb942d2bdb86184be748082" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44753 } }
I20240503 08:07:52.952998 30462 raft_consensus.cc:374] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:52.953015 30462 raft_consensus.cc:729] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b503e70bb942d2bdb86184be748082, State: Initialized, Role: FOLLOWER
I20240503 08:07:52.953052 30462 consensus_queue.cc:260] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b503e70bb942d2bdb86184be748082" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44753 } }
I20240503 08:07:52.953079 30461 tablet_bootstrap.cc:654] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:52.953091 30462 raft_consensus.cc:388] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:52.953114 30462 raft_consensus.cc:482] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:52.953131 30462 raft_consensus.cc:3037] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:52.953558 30461 tablet_bootstrap.cc:492] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786: No bootstrap required, opened a new log
I20240503 08:07:52.953611 30461 ts_tablet_manager.cc:1367] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:52.953652 30462 raft_consensus.cc:504] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b503e70bb942d2bdb86184be748082" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44753 } }
I20240503 08:07:52.953714 30462 leader_election.cc:302] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2b503e70bb942d2bdb86184be748082; no voters:
I20240503 08:07:52.953732 30461 raft_consensus.cc:348] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e661dcbb6467447ca48e2fc307c8b786" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34035 } }
I20240503 08:07:52.953759 30461 raft_consensus.cc:374] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:52.953775 30461 raft_consensus.cc:729] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e661dcbb6467447ca48e2fc307c8b786, State: Initialized, Role: FOLLOWER
I20240503 08:07:52.953806 30461 consensus_queue.cc:260] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e661dcbb6467447ca48e2fc307c8b786" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34035 } }
I20240503 08:07:52.953533 30460 raft_consensus.cc:504] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0e5fe62ac4150912713802b6a4a67" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46177 } }
I20240503 08:07:52.953840 30462 leader_election.cc:288] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:52.953861 30460 leader_election.cc:302] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7a0e5fe62ac4150912713802b6a4a67; no voters:
I20240503 08:07:52.953868 30466 raft_consensus.cc:2781] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:52.953989 30466 raft_consensus.cc:686] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 1 LEADER]: Becoming Leader. State: Replica: a2b503e70bb942d2bdb86184be748082, State: Running, Role: LEADER
I20240503 08:07:52.954046 30466 consensus_queue.cc:237] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b503e70bb942d2bdb86184be748082" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44753 } }
I20240503 08:07:52.954031 30468 raft_consensus.cc:2781] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:52.953843 30461 raft_consensus.cc:388] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:52.954140 30461 raft_consensus.cc:482] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:52.954164 30461 raft_consensus.cc:3037] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:52.954201 30468 raft_consensus.cc:686] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 1 LEADER]: Becoming Leader. State: Replica: f7a0e5fe62ac4150912713802b6a4a67, State: Running, Role: LEADER
I20240503 08:07:52.954250 30468 consensus_queue.cc:237] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0e5fe62ac4150912713802b6a4a67" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46177 } }
I20240503 08:07:52.954548 30102 catalog_manager.cc:5484] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b503e70bb942d2bdb86184be748082 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "a2b503e70bb942d2bdb86184be748082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b503e70bb942d2bdb86184be748082" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44753 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:52.954757 30461 raft_consensus.cc:504] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e661dcbb6467447ca48e2fc307c8b786" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34035 } }
I20240503 08:07:52.954834 30461 leader_election.cc:302] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e661dcbb6467447ca48e2fc307c8b786; no voters:
I20240503 08:07:52.954944 30461 leader_election.cc:288] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:52.954983 30473 raft_consensus.cc:2781] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:52.955063 30461 ts_tablet_manager.cc:1398] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:52.955101 30473 raft_consensus.cc:686] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 1 LEADER]: Becoming Leader. State: Replica: e661dcbb6467447ca48e2fc307c8b786, State: Running, Role: LEADER
I20240503 08:07:52.955157 30473 consensus_queue.cc:237] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e661dcbb6467447ca48e2fc307c8b786" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34035 } }
I20240503 08:07:52.956192 30101 catalog_manager.cc:5484] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 reported cstate change: term changed from 0 to 1, leader changed from <none> to e661dcbb6467447ca48e2fc307c8b786 (127.28.55.66). New cstate: current_term: 1 leader_uuid: "e661dcbb6467447ca48e2fc307c8b786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e661dcbb6467447ca48e2fc307c8b786" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34035 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:52.956779 30100 catalog_manager.cc:5484] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7a0e5fe62ac4150912713802b6a4a67 (127.28.55.65). New cstate: current_term: 1 leader_uuid: "f7a0e5fe62ac4150912713802b6a4a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a0e5fe62ac4150912713802b6a4a67" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46177 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:52.953974 30460 leader_election.cc:288] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:52.957669 30460 ts_tablet_manager.cc:1398] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:07:52.953974 30462 ts_tablet_manager.cc:1398] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:52.963238 28893 master.cc:537] Master@127.28.55.124:39287 shutting down...
I20240503 08:07:52.965339 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:52.965443 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:52.965478 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6cf50db4c4854323a91b6dc5974b3e04: stopping tablet replica
I20240503 08:07:52.978143 28893 master.cc:552] Master@127.28.55.124:39287 shutdown complete.
W20240503 08:07:53.958721 30333 heartbeater.cc:644] Failed to heartbeat to 127.28.55.124:39287 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111)
I20240503 08:07:54.460253 30484 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6cf50db4c4854323a91b6dc5974b3e04)
I20240503 08:07:54.460391 30485 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6cf50db4c4854323a91b6dc5974b3e04)
I20240503 08:07:54.460371 30484 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:54.460536 30485 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:54.460677 30484 leader_election.cc:288] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a47e8670945f441f910030ec750ba530 (127.28.55.126:42917), 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287)
I20240503 08:07:54.460872 29980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab097550721e497e9e163b02cbb702c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a47e8670945f441f910030ec750ba530" is_pre_election: true
I20240503 08:07:54.460882 30046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47e8670945f441f910030ec750ba530" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ab097550721e497e9e163b02cbb702c6" is_pre_election: true
I20240503 08:07:54.460994 29980 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab097550721e497e9e163b02cbb702c6 in term 1.
I20240503 08:07:54.461004 30046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e8670945f441f910030ec750ba530 in term 1.
I20240503 08:07:54.461177 30019 leader_election.cc:302] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e8670945f441f910030ec750ba530, ab097550721e497e9e163b02cbb702c6; no voters:
I20240503 08:07:54.461179 29957 leader_election.cc:302] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e8670945f441f910030ec750ba530, ab097550721e497e9e163b02cbb702c6; no voters:
I20240503 08:07:54.461287 30484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:07:54.461313 30484 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6cf50db4c4854323a91b6dc5974b3e04)
I20240503 08:07:54.461328 30484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:54.461406 30489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:07:54.461452 30489 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6cf50db4c4854323a91b6dc5974b3e04)
I20240503 08:07:54.460894 30485 leader_election.cc:288] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab097550721e497e9e163b02cbb702c6 (127.28.55.125:44695), 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287)
I20240503 08:07:54.461470 30489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:07:54.461663 29955 leader_election.cc:334] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287): Network error: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111)
W20240503 08:07:54.461951 30020 leader_election.cc:334] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287): Network error: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111)
I20240503 08:07:54.462278 30484 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:54.462329 30489 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:54.462383 30484 leader_election.cc:288] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [CANDIDATE]: Term 2 election: Requested vote from peers a47e8670945f441f910030ec750ba530 (127.28.55.126:42917), 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287)
I20240503 08:07:54.462460 30489 leader_election.cc:288] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 2 election: Requested vote from peers ab097550721e497e9e163b02cbb702c6 (127.28.55.125:44695), 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287)
I20240503 08:07:54.462555 29980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab097550721e497e9e163b02cbb702c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "a47e8670945f441f910030ec750ba530"
I20240503 08:07:54.462644 29980 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ab097550721e497e9e163b02cbb702c6 in current term 2: Already voted for candidate a47e8670945f441f910030ec750ba530 in this term.
W20240503 08:07:54.462893 30020 leader_election.cc:334] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287): Network error: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111)
I20240503 08:07:54.462882 30046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47e8670945f441f910030ec750ba530" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ab097550721e497e9e163b02cbb702c6"
I20240503 08:07:54.462965 30020 leader_election.cc:302] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab097550721e497e9e163b02cbb702c6; no voters: 6cf50db4c4854323a91b6dc5974b3e04, a47e8670945f441f910030ec750ba530
I20240503 08:07:54.462983 30046 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a47e8670945f441f910030ec750ba530 in current term 2: Already voted for candidate ab097550721e497e9e163b02cbb702c6 in this term.
W20240503 08:07:54.463014 29955 leader_election.cc:334] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287): Network error: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111)
I20240503 08:07:54.463076 30484 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 08:07:54.463099 29957 leader_election.cc:302] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a47e8670945f441f910030ec750ba530; no voters: 6cf50db4c4854323a91b6dc5974b3e04, ab097550721e497e9e163b02cbb702c6
I20240503 08:07:54.463179 30489 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 08:07:56.050208 30497 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:56.050348 30497 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:56.050566 30497 leader_election.cc:288] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab097550721e497e9e163b02cbb702c6 (127.28.55.125:44695), 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287)
I20240503 08:07:56.050784 30046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47e8670945f441f910030ec750ba530" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ab097550721e497e9e163b02cbb702c6" is_pre_election: true
I20240503 08:07:56.050912 30046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47e8670945f441f910030ec750ba530 in term 2.
I20240503 08:07:56.051074 29957 leader_election.cc:302] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47e8670945f441f910030ec750ba530, ab097550721e497e9e163b02cbb702c6; no voters:
W20240503 08:07:56.051118 29955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 08:07:56.051170 30497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 08:07:56.051227 30497 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:56.051254 30497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:07:56.051636 29955 leader_election.cc:334] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287): Network error: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111)
I20240503 08:07:56.052043 30497 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:56.052181 30497 leader_election.cc:288] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 3 election: Requested vote from peers ab097550721e497e9e163b02cbb702c6 (127.28.55.125:44695), 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287)
I20240503 08:07:56.052337 30046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47e8670945f441f910030ec750ba530" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "ab097550721e497e9e163b02cbb702c6"
I20240503 08:07:56.052402 30046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:07:56.052636 29955 leader_election.cc:334] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287): Network error: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111)
I20240503 08:07:56.053041 30046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47e8670945f441f910030ec750ba530 in term 3.
I20240503 08:07:56.053221 29957 leader_election.cc:302] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a47e8670945f441f910030ec750ba530, ab097550721e497e9e163b02cbb702c6; no voters: 6cf50db4c4854323a91b6dc5974b3e04
I20240503 08:07:56.053314 30497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:07:56.053423 30497 raft_consensus.cc:686] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 3 LEADER]: Becoming Leader. State: Replica: a47e8670945f441f910030ec750ba530, State: Running, Role: LEADER
I20240503 08:07:56.053467 30497 consensus_queue.cc:237] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } }
I20240503 08:07:56.053773 30500 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47e8670945f441f910030ec750ba530. Latest consensus state: current_term: 3 leader_uuid: "a47e8670945f441f910030ec750ba530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:56.053859 30500 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:56.054008 30502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:07:56.054241 30502 catalog_manager.cc:669] Loaded metadata for table test-workload [id=85a640ba7ed646fc88eee63554e38a28]
I20240503 08:07:56.054363 30502 tablet_loader.cc:96] loaded metadata for tablet 92c27a045bf34f30ad68b52fb21cab03 (table test-workload [id=85a640ba7ed646fc88eee63554e38a28])
I20240503 08:07:56.054404 30502 tablet_loader.cc:96] loaded metadata for tablet a507c2f1baf043eca5d1055365ed1824 (table test-workload [id=85a640ba7ed646fc88eee63554e38a28])
I20240503 08:07:56.054437 30502 tablet_loader.cc:96] loaded metadata for tablet d635b046e93b409f91929f465eee4919 (table test-workload [id=85a640ba7ed646fc88eee63554e38a28])
I20240503 08:07:56.054468 30502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:07:56.054546 30502 catalog_manager.cc:1259] Loaded cluster ID: 757b1f287473459f8f66236bab2e4954
I20240503 08:07:56.054571 30502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:07:56.054877 30502 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:07:56.054994 30502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530: Loaded TSK: 0
I20240503 08:07:56.055110 30502 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240503 08:07:56.432708 29955 consensus_peers.cc:485] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 -> Peer 6cf50db4c4854323a91b6dc5974b3e04 (127.28.55.124:39287): Couldn't send request to peer 6cf50db4c4854323a91b6dc5974b3e04. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.124:39287: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:07:56.544781 30154 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:07:56.544903 30154 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:07:56.599350 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:07:56.602015 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:56.602147 28893 tablet_replica.cc:330] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67: stopping tablet replica
I20240503 08:07:56.602205 28893 raft_consensus.cc:2229] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:56.602257 28893 raft_consensus.cc:2258] T 92c27a045bf34f30ad68b52fb21cab03 P f7a0e5fe62ac4150912713802b6a4a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:56.614034 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:07:56.615083 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:07:56.615805 30046 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 3 FOLLOWER]: Refusing update from remote peer a47e8670945f441f910030ec750ba530: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 08:07:56.616147 30500 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:07:56.616724 30503 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47e8670945f441f910030ec750ba530. Latest consensus state: current_term: 3 leader_uuid: "a47e8670945f441f910030ec750ba530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:56.617097 30503 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:56.617365 30500 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a47e8670945f441f910030ec750ba530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:56.617437 30500 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:56.618072 30503 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a47e8670945f441f910030ec750ba530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47e8670945f441f910030ec750ba530" member_type: VOTER last_known_addr { host: "127.28.55.126" port: 42917 } } peers { permanent_uuid: "ab097550721e497e9e163b02cbb702c6" member_type: VOTER last_known_addr { host: "127.28.55.125" port: 44695 } } peers { permanent_uuid: "6cf50db4c4854323a91b6dc5974b3e04" member_type: VOTER last_known_addr { host: "127.28.55.124" port: 39287 } } }
I20240503 08:07:56.618171 30503 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 08:07:56.618382 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:56.618495 28893 tablet_replica.cc:330] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786: stopping tablet replica
I20240503 08:07:56.618556 28893 raft_consensus.cc:2229] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:56.618588 28893 raft_consensus.cc:2258] T a507c2f1baf043eca5d1055365ed1824 P e661dcbb6467447ca48e2fc307c8b786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:56.630183 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:07:56.631120 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:07:56.633762 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:56.633884 28893 tablet_replica.cc:330] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082: stopping tablet replica
I20240503 08:07:56.633931 28893 raft_consensus.cc:2229] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:56.633985 28893 raft_consensus.cc:2258] T d635b046e93b409f91929f465eee4919 P a2b503e70bb942d2bdb86184be748082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:56.645654 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:07:56.646535 28893 master.cc:537] Master@127.28.55.126:42917 shutting down...
I20240503 08:07:56.648656 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:07:56.648756 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:56.648823 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P a47e8670945f441f910030ec750ba530: stopping tablet replica
I20240503 08:07:56.660992 28893 master.cc:552] Master@127.28.55.126:42917 shutdown complete.
I20240503 08:07:56.662267 28893 master.cc:537] Master@127.28.55.125:44695 shutting down...
I20240503 08:07:56.664666 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:56.664747 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:56.664769 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P ab097550721e497e9e163b02cbb702c6: stopping tablet replica
I20240503 08:07:56.676805 28893 master.cc:552] Master@127.28.55.125:44695 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6206 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 08:07:56.682015 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:44463
I20240503 08:07:56.682188 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:56.683485 30509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:56.683559 30510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:56.683672 30512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:56.683691 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:56.683877 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:56.683921 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:56.683935 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723676683936 us; error 0 us; skew 500 ppm
I20240503 08:07:56.688344 28893 webserver.cc:460] Webserver started at http://127.28.55.126:46343/ using document root <none> and password file <none>
I20240503 08:07:56.688434 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:56.688480 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:56.688537 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:56.688818 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "78c68782ba6448b7b462071e63b46452"
format_stamp: "Formatted at 2024-05-03 08:07:56 on dist-test-slave-mlgg"
I20240503 08:07:56.689677 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:56.690128 30517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.690253 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.690295 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "78c68782ba6448b7b462071e63b46452"
format_stamp: "Formatted at 2024-05-03 08:07:56 on dist-test-slave-mlgg"
I20240503 08:07:56.690343 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:56.721158 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:56.721392 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:56.724010 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:44463
I20240503 08:07:56.726445 30569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:44463 every 8 connection(s)
I20240503 08:07:56.726622 30570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:56.727705 30570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452: Bootstrap starting.
I20240503 08:07:56.727957 30570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:56.728454 30570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452: No bootstrap required, opened a new log
I20240503 08:07:56.728575 30570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c68782ba6448b7b462071e63b46452" member_type: VOTER }
I20240503 08:07:56.728627 30570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:56.728638 30570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c68782ba6448b7b462071e63b46452, State: Initialized, Role: FOLLOWER
I20240503 08:07:56.728675 30570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c68782ba6448b7b462071e63b46452" member_type: VOTER }
I20240503 08:07:56.728713 30570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:56.728729 30570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:56.728754 30570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:56.729215 30570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c68782ba6448b7b462071e63b46452" member_type: VOTER }
I20240503 08:07:56.729274 30570 leader_election.cc:302] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78c68782ba6448b7b462071e63b46452; no voters:
I20240503 08:07:56.729372 30570 leader_election.cc:288] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:56.729440 30573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:56.729501 30570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:56.729584 30573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 1 LEADER]: Becoming Leader. State: Replica: 78c68782ba6448b7b462071e63b46452, State: Running, Role: LEADER
I20240503 08:07:56.729645 30573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c68782ba6448b7b462071e63b46452" member_type: VOTER }
I20240503 08:07:56.729866 30574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78c68782ba6448b7b462071e63b46452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c68782ba6448b7b462071e63b46452" member_type: VOTER } }
I20240503 08:07:56.729878 30575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78c68782ba6448b7b462071e63b46452. Latest consensus state: current_term: 1 leader_uuid: "78c68782ba6448b7b462071e63b46452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c68782ba6448b7b462071e63b46452" member_type: VOTER } }
I20240503 08:07:56.729955 30574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:56.729961 30575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:56.730212 30579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:07:56.730352 30579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:07:56.730785 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:07:56.730968 30579 catalog_manager.cc:1347] Generated new cluster ID: e2b60886d9f74e709fe7b6f60441c459
I20240503 08:07:56.731015 30579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:07:56.755918 30579 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:07:56.756393 30579 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:07:56.760537 30579 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452: Generated new TSK 0
I20240503 08:07:56.760629 30579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:07:56.762735 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:56.764144 30592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:56.764286 30594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:56.764305 30591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:56.764358 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:56.764447 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:56.764477 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:56.764501 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723676764501 us; error 0 us; skew 500 ppm
I20240503 08:07:56.765184 28893 webserver.cc:460] Webserver started at http://127.28.55.65:34559/ using document root <none> and password file <none>
I20240503 08:07:56.765260 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:56.765317 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:56.765364 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:56.765668 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "43a08d6481084df8a886fafd6a2bfdde"
format_stamp: "Formatted at 2024-05-03 08:07:56 on dist-test-slave-mlgg"
I20240503 08:07:56.766556 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:56.767127 30599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.767266 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.767309 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "43a08d6481084df8a886fafd6a2bfdde"
format_stamp: "Formatted at 2024-05-03 08:07:56 on dist-test-slave-mlgg"
I20240503 08:07:56.767365 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:56.787184 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:56.787420 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:56.787585 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:56.787786 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:56.787814 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.787844 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:56.787868 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.790650 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:46553
I20240503 08:07:56.790699 30662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:46553 every 8 connection(s)
I20240503 08:07:56.791069 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:56.792276 30667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:56.792313 28893 server_base.cc:1034] running on GCE node
W20240503 08:07:56.792451 30670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:56.793179 30668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:56.793365 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:56.793401 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:56.793418 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723676793418 us; error 0 us; skew 500 ppm
I20240503 08:07:56.794174 28893 webserver.cc:460] Webserver started at http://127.28.55.66:42437/ using document root <none> and password file <none>
I20240503 08:07:56.794265 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:56.794312 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:56.794361 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:56.794657 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "a3dc67979925427dae2127e8f527761b"
format_stamp: "Formatted at 2024-05-03 08:07:56 on dist-test-slave-mlgg"
I20240503 08:07:56.795156 30663 heartbeater.cc:347] Connected to a master server at 127.28.55.126:44463
I20240503 08:07:56.795212 30663 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:56.795316 30663 heartbeater.cc:511] Master 127.28.55.126:44463 requested a full tablet report, sending...
I20240503 08:07:56.795545 30534 ts_manager.cc:194] Registered new tserver with Master: 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65:46553)
I20240503 08:07:56.795630 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:56.796113 30675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.796149 30534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51488
I20240503 08:07:56.796229 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.796273 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "a3dc67979925427dae2127e8f527761b"
format_stamp: "Formatted at 2024-05-03 08:07:56 on dist-test-slave-mlgg"
I20240503 08:07:56.796324 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:56.801851 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:56.802045 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:56.802181 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:56.802371 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:56.802398 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.802426 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:56.802450 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.805070 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:42657
I20240503 08:07:56.805135 30738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:42657 every 8 connection(s)
I20240503 08:07:56.805428 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:56.806496 30742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:56.806515 30745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:56.806602 30743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:56.806946 28893 server_base.cc:1034] running on GCE node
I20240503 08:07:56.807082 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:56.807112 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:56.807137 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723676807136 us; error 0 us; skew 500 ppm
I20240503 08:07:56.807988 28893 webserver.cc:460] Webserver started at http://127.28.55.67:34909/ using document root <none> and password file <none>
I20240503 08:07:56.808122 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:56.808173 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:56.808219 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:56.808418 30739 heartbeater.cc:347] Connected to a master server at 127.28.55.126:44463
I20240503 08:07:56.808471 30739 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:56.808564 30739 heartbeater.cc:511] Master 127.28.55.126:44463 requested a full tablet report, sending...
I20240503 08:07:56.808583 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "6d678e76f65343b59f26d383e881aeea"
format_stamp: "Formatted at 2024-05-03 08:07:56 on dist-test-slave-mlgg"
I20240503 08:07:56.808816 30534 ts_manager.cc:194] Registered new tserver with Master: a3dc67979925427dae2127e8f527761b (127.28.55.66:42657)
I20240503 08:07:56.809273 30534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51498
I20240503 08:07:56.809638 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:07:56.810078 30750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.810207 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.810262 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "6d678e76f65343b59f26d383e881aeea"
format_stamp: "Formatted at 2024-05-03 08:07:56 on dist-test-slave-mlgg"
I20240503 08:07:56.810303 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:56.838948 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:56.839182 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:56.839321 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:56.839527 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:56.839555 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.839586 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:56.839609 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:56.842710 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:44509
I20240503 08:07:56.842823 30813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:44509 every 8 connection(s)
I20240503 08:07:56.845818 30814 heartbeater.cc:347] Connected to a master server at 127.28.55.126:44463
I20240503 08:07:56.845885 30814 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:56.845985 30814 heartbeater.cc:511] Master 127.28.55.126:44463 requested a full tablet report, sending...
I20240503 08:07:56.846330 30534 ts_manager.cc:194] Registered new tserver with Master: 6d678e76f65343b59f26d383e881aeea (127.28.55.67:44509)
I20240503 08:07:56.846678 30534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51514
I20240503 08:07:56.847368 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004467625s
I20240503 08:07:57.796659 30663 heartbeater.cc:503] Master 127.28.55.126:44463 was elected leader, sending a full tablet report...
I20240503 08:07:57.809823 30739 heartbeater.cc:503] Master 127.28.55.126:44463 was elected leader, sending a full tablet report...
I20240503 08:07:57.847240 30814 heartbeater.cc:503] Master 127.28.55.126:44463 was elected leader, sending a full tablet report...
I20240503 08:07:57.872680 28893 test_util.cc:268] Using random seed: 934617636
I20240503 08:07:57.878336 30534 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36692:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:07:57.878494 30534 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:07:57.884012 30627 tablet_service.cc:1467] Processing CreateTablet for tablet 245abe56ce504129b23b513ed0f62565 (DEFAULT_TABLE table=test-workload [id=9c8d26ee8f0449dc9b45e732d00e5c58]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 08:07:57.884145 30703 tablet_service.cc:1467] Processing CreateTablet for tablet 245abe56ce504129b23b513ed0f62565 (DEFAULT_TABLE table=test-workload [id=9c8d26ee8f0449dc9b45e732d00e5c58]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 08:07:57.884235 30702 tablet_service.cc:1467] Processing CreateTablet for tablet 3cff297fd92a4b218c50a228ae8ac908 (DEFAULT_TABLE table=test-workload [id=9c8d26ee8f0449dc9b45e732d00e5c58]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 08:07:57.884258 30626 tablet_service.cc:1467] Processing CreateTablet for tablet 3cff297fd92a4b218c50a228ae8ac908 (DEFAULT_TABLE table=test-workload [id=9c8d26ee8f0449dc9b45e732d00e5c58]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 08:07:57.884285 30703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245abe56ce504129b23b513ed0f62565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:57.884393 30702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cff297fd92a4b218c50a228ae8ac908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:57.884483 30778 tablet_service.cc:1467] Processing CreateTablet for tablet 245abe56ce504129b23b513ed0f62565 (DEFAULT_TABLE table=test-workload [id=9c8d26ee8f0449dc9b45e732d00e5c58]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 08:07:57.884557 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245abe56ce504129b23b513ed0f62565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:57.885545 30834 tablet_bootstrap.cc:492] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b: Bootstrap starting.
I20240503 08:07:57.885550 30777 tablet_service.cc:1467] Processing CreateTablet for tablet 3cff297fd92a4b218c50a228ae8ac908 (DEFAULT_TABLE table=test-workload [id=9c8d26ee8f0449dc9b45e732d00e5c58]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 08:07:57.885841 30835 tablet_bootstrap.cc:492] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea: Bootstrap starting.
I20240503 08:07:57.885867 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cff297fd92a4b218c50a228ae8ac908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:57.885958 30834 tablet_bootstrap.cc:654] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:57.886384 30835 tablet_bootstrap.cc:654] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:57.886577 30834 tablet_bootstrap.cc:492] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b: No bootstrap required, opened a new log
I20240503 08:07:57.886618 30834 ts_tablet_manager.cc:1367] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:57.886763 30834 raft_consensus.cc:348] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } }
I20240503 08:07:57.886857 30834 raft_consensus.cc:374] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:57.886878 30834 raft_consensus.cc:729] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dc67979925427dae2127e8f527761b, State: Initialized, Role: FOLLOWER
I20240503 08:07:57.886921 30834 consensus_queue.cc:260] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } }
I20240503 08:07:57.886945 30835 tablet_bootstrap.cc:492] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea: No bootstrap required, opened a new log
I20240503 08:07:57.886986 30835 ts_tablet_manager.cc:1367] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:57.887079 30834 ts_tablet_manager.cc:1398] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:57.887146 30834 tablet_bootstrap.cc:492] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b: Bootstrap starting.
I20240503 08:07:57.887125 30835 raft_consensus.cc:348] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } }
I20240503 08:07:57.887183 30835 raft_consensus.cc:374] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:57.887202 30835 raft_consensus.cc:729] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d678e76f65343b59f26d383e881aeea, State: Initialized, Role: FOLLOWER
I20240503 08:07:57.887264 30835 consensus_queue.cc:260] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } }
I20240503 08:07:57.887406 30835 ts_tablet_manager.cc:1398] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:57.887463 30835 tablet_bootstrap.cc:492] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea: Bootstrap starting.
I20240503 08:07:57.887504 30834 tablet_bootstrap.cc:654] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:57.887903 30835 tablet_bootstrap.cc:654] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:57.884189 30627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245abe56ce504129b23b513ed0f62565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:57.888188 30626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cff297fd92a4b218c50a228ae8ac908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:57.888211 30834 tablet_bootstrap.cc:492] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b: No bootstrap required, opened a new log
I20240503 08:07:57.888345 30834 ts_tablet_manager.cc:1367] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:57.888531 30834 raft_consensus.cc:348] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:57.888593 30834 raft_consensus.cc:374] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:57.888610 30834 raft_consensus.cc:729] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dc67979925427dae2127e8f527761b, State: Initialized, Role: FOLLOWER
I20240503 08:07:57.888670 30834 consensus_queue.cc:260] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:57.888839 30834 ts_tablet_manager.cc:1398] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:57.889554 30842 tablet_bootstrap.cc:492] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde: Bootstrap starting.
I20240503 08:07:57.889780 30835 tablet_bootstrap.cc:492] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea: No bootstrap required, opened a new log
I20240503 08:07:57.889822 30835 ts_tablet_manager.cc:1367] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:07:57.889925 30842 tablet_bootstrap.cc:654] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:57.889961 30835 raft_consensus.cc:348] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:57.890018 30835 raft_consensus.cc:374] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:57.890044 30835 raft_consensus.cc:729] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d678e76f65343b59f26d383e881aeea, State: Initialized, Role: FOLLOWER
I20240503 08:07:57.890094 30835 consensus_queue.cc:260] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:57.890175 30835 ts_tablet_manager.cc:1398] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:57.890488 30842 tablet_bootstrap.cc:492] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde: No bootstrap required, opened a new log
I20240503 08:07:57.890549 30842 ts_tablet_manager.cc:1367] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:57.890702 30842 raft_consensus.cc:348] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } }
I20240503 08:07:57.890767 30842 raft_consensus.cc:374] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:57.890792 30842 raft_consensus.cc:729] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a08d6481084df8a886fafd6a2bfdde, State: Initialized, Role: FOLLOWER
I20240503 08:07:57.890872 30842 consensus_queue.cc:260] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } }
I20240503 08:07:57.891041 30842 ts_tablet_manager.cc:1398] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:57.891103 30842 tablet_bootstrap.cc:492] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde: Bootstrap starting.
I20240503 08:07:57.891427 30842 tablet_bootstrap.cc:654] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:57.891919 30842 tablet_bootstrap.cc:492] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde: No bootstrap required, opened a new log
I20240503 08:07:57.891978 30842 ts_tablet_manager.cc:1367] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:57.892093 30842 raft_consensus.cc:348] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:57.892139 30842 raft_consensus.cc:374] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:57.892150 30842 raft_consensus.cc:729] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a08d6481084df8a886fafd6a2bfdde, State: Initialized, Role: FOLLOWER
I20240503 08:07:57.892197 30842 consensus_queue.cc:260] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:57.892290 30842 ts_tablet_manager.cc:1398] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:58.094779 30839 raft_consensus.cc:482] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:58.094936 30839 raft_consensus.cc:504] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:58.095194 30839 leader_election.cc:288] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65:46553), a3dc67979925427dae2127e8f527761b (127.28.55.66:42657)
I20240503 08:07:58.098950 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff297fd92a4b218c50a228ae8ac908" candidate_uuid: "6d678e76f65343b59f26d383e881aeea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a08d6481084df8a886fafd6a2bfdde" is_pre_election: true
I20240503 08:07:58.098950 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff297fd92a4b218c50a228ae8ac908" candidate_uuid: "6d678e76f65343b59f26d383e881aeea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dc67979925427dae2127e8f527761b" is_pre_election: true
I20240503 08:07:58.099121 30637 raft_consensus.cc:2454] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d678e76f65343b59f26d383e881aeea in term 0.
I20240503 08:07:58.099121 30713 raft_consensus.cc:2454] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d678e76f65343b59f26d383e881aeea in term 0.
I20240503 08:07:58.099316 30752 leader_election.cc:302] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43a08d6481084df8a886fafd6a2bfdde, 6d678e76f65343b59f26d383e881aeea, a3dc67979925427dae2127e8f527761b; no voters:
I20240503 08:07:58.099416 30839 raft_consensus.cc:2781] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:58.099453 30839 raft_consensus.cc:482] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:58.099479 30839 raft_consensus.cc:3037] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:58.100152 30839 raft_consensus.cc:504] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:58.100288 30839 leader_election.cc:288] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [CANDIDATE]: Term 1 election: Requested vote from peers 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65:46553), a3dc67979925427dae2127e8f527761b (127.28.55.66:42657)
I20240503 08:07:58.100471 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff297fd92a4b218c50a228ae8ac908" candidate_uuid: "6d678e76f65343b59f26d383e881aeea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dc67979925427dae2127e8f527761b"
I20240503 08:07:58.100471 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff297fd92a4b218c50a228ae8ac908" candidate_uuid: "6d678e76f65343b59f26d383e881aeea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a08d6481084df8a886fafd6a2bfdde"
I20240503 08:07:58.100579 30713 raft_consensus.cc:3037] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:58.100584 30637 raft_consensus.cc:3037] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:58.101311 30713 raft_consensus.cc:2454] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d678e76f65343b59f26d383e881aeea in term 1.
I20240503 08:07:58.101311 30637 raft_consensus.cc:2454] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d678e76f65343b59f26d383e881aeea in term 1.
I20240503 08:07:58.101537 30752 leader_election.cc:302] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a08d6481084df8a886fafd6a2bfdde, 6d678e76f65343b59f26d383e881aeea; no voters:
I20240503 08:07:58.101615 30839 raft_consensus.cc:2781] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:58.101751 30839 raft_consensus.cc:686] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Becoming Leader. State: Replica: 6d678e76f65343b59f26d383e881aeea, State: Running, Role: LEADER
I20240503 08:07:58.101810 30839 consensus_queue.cc:237] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:58.102396 30533 catalog_manager.cc:5484] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d678e76f65343b59f26d383e881aeea (127.28.55.67). New cstate: current_term: 1 leader_uuid: "6d678e76f65343b59f26d383e881aeea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:58.145190 30839 raft_consensus.cc:482] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:58.145313 30839 raft_consensus.cc:504] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } }
I20240503 08:07:58.145516 30839 leader_election.cc:288] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65:46553), a3dc67979925427dae2127e8f527761b (127.28.55.66:42657)
I20240503 08:07:58.145766 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245abe56ce504129b23b513ed0f62565" candidate_uuid: "6d678e76f65343b59f26d383e881aeea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a08d6481084df8a886fafd6a2bfdde" is_pre_election: true
I20240503 08:07:58.145768 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245abe56ce504129b23b513ed0f62565" candidate_uuid: "6d678e76f65343b59f26d383e881aeea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dc67979925427dae2127e8f527761b" is_pre_election: true
I20240503 08:07:58.145867 30637 raft_consensus.cc:2454] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d678e76f65343b59f26d383e881aeea in term 0.
I20240503 08:07:58.145888 30713 raft_consensus.cc:2454] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d678e76f65343b59f26d383e881aeea in term 0.
I20240503 08:07:58.146036 30751 leader_election.cc:302] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d678e76f65343b59f26d383e881aeea, a3dc67979925427dae2127e8f527761b; no voters:
I20240503 08:07:58.146129 30839 raft_consensus.cc:2781] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:58.146165 30839 raft_consensus.cc:482] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:58.146178 30839 raft_consensus.cc:3037] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:58.146798 30839 raft_consensus.cc:504] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } }
I20240503 08:07:58.146953 30839 leader_election.cc:288] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [CANDIDATE]: Term 1 election: Requested vote from peers 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65:46553), a3dc67979925427dae2127e8f527761b (127.28.55.66:42657)
I20240503 08:07:58.147118 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245abe56ce504129b23b513ed0f62565" candidate_uuid: "6d678e76f65343b59f26d383e881aeea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dc67979925427dae2127e8f527761b"
I20240503 08:07:58.147130 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245abe56ce504129b23b513ed0f62565" candidate_uuid: "6d678e76f65343b59f26d383e881aeea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a08d6481084df8a886fafd6a2bfdde"
I20240503 08:07:58.147181 30713 raft_consensus.cc:3037] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:58.147214 30637 raft_consensus.cc:3037] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:58.147734 30713 raft_consensus.cc:2454] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d678e76f65343b59f26d383e881aeea in term 1.
I20240503 08:07:58.147806 30637 raft_consensus.cc:2454] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d678e76f65343b59f26d383e881aeea in term 1.
I20240503 08:07:58.147905 30751 leader_election.cc:302] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d678e76f65343b59f26d383e881aeea, a3dc67979925427dae2127e8f527761b; no voters:
I20240503 08:07:58.148038 30839 raft_consensus.cc:2781] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:58.148079 30839 raft_consensus.cc:686] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Becoming Leader. State: Replica: 6d678e76f65343b59f26d383e881aeea, State: Running, Role: LEADER
I20240503 08:07:58.148135 30839 consensus_queue.cc:237] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } }
I20240503 08:07:58.148797 30533 catalog_manager.cc:5484] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d678e76f65343b59f26d383e881aeea (127.28.55.67). New cstate: current_term: 1 leader_uuid: "6d678e76f65343b59f26d383e881aeea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:58.524667 30839 consensus_queue.cc:1035] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:58.526386 30849 consensus_queue.cc:1035] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:58.655370 30849 consensus_queue.cc:1035] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:58.656772 30849 consensus_queue.cc:1035] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:58.730896 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:07:58.733918 30788 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3cff297fd92a4b218c50a228ae8ac908"
dest_uuid: "6d678e76f65343b59f26d383e881aeea"
mode: GRACEFUL
new_leader_uuid: "43a08d6481084df8a886fafd6a2bfdde"
from {username='slave'} at 127.0.0.1:55088
I20240503 08:07:58.734020 30788 raft_consensus.cc:595] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Received request to transfer leadership to TS 43a08d6481084df8a886fafd6a2bfdde
I20240503 08:07:58.734294 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:07:58.734359 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:07:59.038915 30839 raft_consensus.cc:979] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea: : Instructing follower 43a08d6481084df8a886fafd6a2bfdde to start an election
I20240503 08:07:59.039023 30839 raft_consensus.cc:1067] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Signalling peer 43a08d6481084df8a886fafd6a2bfdde to start an election
I20240503 08:07:59.039223 30637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3cff297fd92a4b218c50a228ae8ac908"
dest_uuid: "43a08d6481084df8a886fafd6a2bfdde"
from {username='slave'} at 127.0.0.1:45578
I20240503 08:07:59.039319 30637 raft_consensus.cc:482] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:07:59.039348 30637 raft_consensus.cc:3037] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:59.039996 30637 raft_consensus.cc:504] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:59.040171 30637 leader_election.cc:288] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [CANDIDATE]: Term 2 election: Requested vote from peers a3dc67979925427dae2127e8f527761b (127.28.55.66:42657), 6d678e76f65343b59f26d383e881aeea (127.28.55.67:44509)
I20240503 08:07:59.043555 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff297fd92a4b218c50a228ae8ac908" candidate_uuid: "43a08d6481084df8a886fafd6a2bfdde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6d678e76f65343b59f26d383e881aeea"
I20240503 08:07:59.043663 30788 raft_consensus.cc:3032] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:07:59.043673 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cff297fd92a4b218c50a228ae8ac908" candidate_uuid: "43a08d6481084df8a886fafd6a2bfdde" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3dc67979925427dae2127e8f527761b"
I20240503 08:07:59.043681 30788 raft_consensus.cc:729] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d678e76f65343b59f26d383e881aeea, State: Running, Role: LEADER
I20240503 08:07:59.043766 30713 raft_consensus.cc:3037] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:59.043812 30788 consensus_queue.cc:260] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:59.043908 30788 raft_consensus.cc:3037] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:59.044428 30713 raft_consensus.cc:2454] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a08d6481084df8a886fafd6a2bfdde in term 2.
I20240503 08:07:59.044494 30788 raft_consensus.cc:2454] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a08d6481084df8a886fafd6a2bfdde in term 2.
I20240503 08:07:59.044632 30600 leader_election.cc:302] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a08d6481084df8a886fafd6a2bfdde, a3dc67979925427dae2127e8f527761b; no voters:
I20240503 08:07:59.044761 30856 raft_consensus.cc:2781] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:07:59.044907 30856 raft_consensus.cc:686] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 2 LEADER]: Becoming Leader. State: Replica: 43a08d6481084df8a886fafd6a2bfdde, State: Running, Role: LEADER
I20240503 08:07:59.044968 30856 consensus_queue.cc:237] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } }
I20240503 08:07:59.045419 30533 catalog_manager.cc:5484] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde reported cstate change: term changed from 1 to 2, leader changed from 6d678e76f65343b59f26d383e881aeea (127.28.55.67) to 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65). New cstate: current_term: 2 leader_uuid: "43a08d6481084df8a886fafd6a2bfdde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:59.515311 30713 raft_consensus.cc:1261] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 2 FOLLOWER]: Refusing update from remote peer 43a08d6481084df8a886fafd6a2bfdde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:59.515628 30856 consensus_queue.cc:1035] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:07:59.516793 30788 raft_consensus.cc:1261] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 2 FOLLOWER]: Refusing update from remote peer 43a08d6481084df8a886fafd6a2bfdde: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:07:59.516981 30856 consensus_queue.cc:1035] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:08:00.734548 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:00.734735 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:00.734779 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:02.734982 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:02.735159 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:02.735198 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:04.735392 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:04.735558 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:04.735600 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:06.735754 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:06.735901 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:06.735934 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:08.736122 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:08.736302 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:08.736327 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:09.216040 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:09.217772 30881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:09.217846 28893 server_base.cc:1034] running on GCE node
W20240503 08:08:09.217903 30879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:09.217918 30878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:09.218210 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:09.218245 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:09.218261 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723689218262 us; error 0 us; skew 500 ppm
I20240503 08:08:09.219086 28893 webserver.cc:460] Webserver started at http://127.28.55.68:43413/ using document root <none> and password file <none>
I20240503 08:08:09.219174 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:09.219225 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:09.219275 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:09.219569 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-3-root/instance:
uuid: "2774224e751644b5810fcb39969284e9"
format_stamp: "Formatted at 2024-05-03 08:08:09 on dist-test-slave-mlgg"
I20240503 08:08:09.220439 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:09.220896 30886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:09.221045 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:09.221086 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-3-root
uuid: "2774224e751644b5810fcb39969284e9"
format_stamp: "Formatted at 2024-05-03 08:08:09 on dist-test-slave-mlgg"
I20240503 08:08:09.221129 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:09.236060 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:09.236347 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:09.236524 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:09.236747 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:09.236783 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:09.236804 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:09.236815 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:09.240089 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.68:43147
I20240503 08:08:09.240139 30949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.68:43147 every 8 connection(s)
I20240503 08:08:09.243481 30950 heartbeater.cc:347] Connected to a master server at 127.28.55.126:44463
I20240503 08:08:09.243557 30950 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:09.243674 30950 heartbeater.cc:511] Master 127.28.55.126:44463 requested a full tablet report, sending...
I20240503 08:08:09.243937 30534 ts_manager.cc:194] Registered new tserver with Master: 2774224e751644b5810fcb39969284e9 (127.28.55.68:43147)
I20240503 08:08:09.244330 30534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35662
I20240503 08:08:09.737401 30788 consensus_queue.cc:237] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: true } }
I20240503 08:08:09.738292 30637 raft_consensus.cc:1261] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Refusing update from remote peer 6d678e76f65343b59f26d383e881aeea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:09.738298 30713 raft_consensus.cc:1261] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Refusing update from remote peer 6d678e76f65343b59f26d383e881aeea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:09.738505 30957 consensus_queue.cc:1035] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:08:09.738677 30958 consensus_queue.cc:1035] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:08:09.739338 30957 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2774224e751644b5810fcb39969284e9 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: true } } }
I20240503 08:08:09.739454 30637 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2774224e751644b5810fcb39969284e9 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: true } } }
I20240503 08:08:09.740368 30713 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2774224e751644b5810fcb39969284e9 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: true } } }
I20240503 08:08:09.740708 30533 catalog_manager.cc:5484] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea reported cstate change: config changed from index -1 to 2, NON_VOTER 2774224e751644b5810fcb39969284e9 (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "6d678e76f65343b59f26d383e881aeea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:08:09.742429 30752 consensus_peers.cc:485] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea -> Peer 2774224e751644b5810fcb39969284e9 (127.28.55.68:43147): Couldn't send request to peer 2774224e751644b5810fcb39969284e9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 245abe56ce504129b23b513ed0f62565. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:08:10.229041 30968 ts_tablet_manager.cc:902] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: Initiating tablet copy from peer 6d678e76f65343b59f26d383e881aeea (127.28.55.67:44509)
I20240503 08:08:10.229525 30968 tablet_copy_client.cc:308] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.67:44509
I20240503 08:08:10.233168 30798 tablet_copy_service.cc:140] P 6d678e76f65343b59f26d383e881aeea: Received BeginTabletCopySession request for tablet 245abe56ce504129b23b513ed0f62565 from peer 2774224e751644b5810fcb39969284e9 ({username='slave'} at 127.0.0.1:51154)
I20240503 08:08:10.233268 30798 tablet_copy_service.cc:161] P 6d678e76f65343b59f26d383e881aeea: Beginning new tablet copy session on tablet 245abe56ce504129b23b513ed0f62565 from peer 2774224e751644b5810fcb39969284e9 at {username='slave'} at 127.0.0.1:51154: session id = 2774224e751644b5810fcb39969284e9-245abe56ce504129b23b513ed0f62565
I20240503 08:08:10.234032 30798 tablet_copy_source_session.cc:205] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:08:10.234617 30968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245abe56ce504129b23b513ed0f62565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:10.235996 30968 tablet_copy_client.cc:783] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: tablet copy: Starting download of 0 data blocks...
I20240503 08:08:10.236141 30968 tablet_copy_client.cc:647] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:08:10.236923 30968 tablet_copy_client.cc:523] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:08:10.237618 30968 tablet_bootstrap.cc:492] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: Bootstrap starting.
I20240503 08:08:10.239094 30968 tablet_bootstrap.cc:492] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:08:10.239288 30968 tablet_bootstrap.cc:492] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: Bootstrap complete.
I20240503 08:08:10.239388 30968 ts_tablet_manager.cc:1367] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:08:10.239558 30968 raft_consensus.cc:348] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: true } }
I20240503 08:08:10.239626 30968 raft_consensus.cc:729] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2774224e751644b5810fcb39969284e9, State: Initialized, Role: LEARNER
I20240503 08:08:10.239660 30968 consensus_queue.cc:260] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: true } }
I20240503 08:08:10.239846 30968 ts_tablet_manager.cc:1398] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:08:10.239881 30950 heartbeater.cc:503] Master 127.28.55.126:44463 was elected leader, sending a full tablet report...
I20240503 08:08:10.240198 30798 tablet_copy_service.cc:342] P 6d678e76f65343b59f26d383e881aeea: Request end of tablet copy session 2774224e751644b5810fcb39969284e9-245abe56ce504129b23b513ed0f62565 received from {username='slave'} at 127.0.0.1:51154
I20240503 08:08:10.240263 30798 tablet_copy_service.cc:433] P 6d678e76f65343b59f26d383e881aeea: ending tablet copy session 2774224e751644b5810fcb39969284e9-245abe56ce504129b23b513ed0f62565 on tablet 245abe56ce504129b23b513ed0f62565 with peer 2774224e751644b5810fcb39969284e9
I20240503 08:08:10.736471 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:10.736613 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:10.736649 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:10.766767 30924 raft_consensus.cc:1203] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:08:11.164700 30976 raft_consensus.cc:1050] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea: attempting to promote NON_VOTER 2774224e751644b5810fcb39969284e9 to VOTER
I20240503 08:08:11.164994 30976 consensus_queue.cc:237] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } }
I20240503 08:08:11.165603 30924 raft_consensus.cc:1261] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [term 1 LEARNER]: Refusing update from remote peer 6d678e76f65343b59f26d383e881aeea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:08:11.165597 30713 raft_consensus.cc:1261] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Refusing update from remote peer 6d678e76f65343b59f26d383e881aeea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:08:11.165870 30637 raft_consensus.cc:1261] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Refusing update from remote peer 6d678e76f65343b59f26d383e881aeea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:08:11.165868 30980 consensus_queue.cc:1035] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:08:11.166038 30979 consensus_queue.cc:1035] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:08:11.166191 30976 consensus_queue.cc:1035] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:08:11.167030 30980 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2774224e751644b5810fcb39969284e9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } } }
I20240503 08:08:11.167141 30713 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2774224e751644b5810fcb39969284e9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } } }
I20240503 08:08:11.167230 30924 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2774224e751644b5810fcb39969284e9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } } }
I20240503 08:08:11.168334 30533 catalog_manager.cc:5484] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 reported cstate change: config changed from index 2 to 3, 2774224e751644b5810fcb39969284e9 (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6d678e76f65343b59f26d383e881aeea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } } }
I20240503 08:08:11.168854 30637 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2774224e751644b5810fcb39969284e9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43a08d6481084df8a886fafd6a2bfdde" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46553 } attrs { replace: true } } peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } } }
I20240503 08:08:11.170482 30788 consensus_queue.cc:237] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } }
I20240503 08:08:11.170984 30924 raft_consensus.cc:1261] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [term 1 FOLLOWER]: Refusing update from remote peer 6d678e76f65343b59f26d383e881aeea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:08:11.171097 30713 raft_consensus.cc:1261] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Refusing update from remote peer 6d678e76f65343b59f26d383e881aeea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:08:11.171171 30979 consensus_queue.cc:1035] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:08:11.171288 30957 consensus_queue.cc:1035] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:08:11.171859 30957 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } } }
I20240503 08:08:11.172044 30713 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } } }
I20240503 08:08:11.172852 30519 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 245abe56ce504129b23b513ed0f62565 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:08:11.172945 30534 catalog_manager.cc:5484] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b reported cstate change: config changed from index 3 to 4, VOTER 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65) evicted. New cstate: current_term: 1 leader_uuid: "6d678e76f65343b59f26d383e881aeea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } } }
I20240503 08:08:11.173092 30924 raft_consensus.cc:2932] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d678e76f65343b59f26d383e881aeea" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44509 } } peers { permanent_uuid: "a3dc67979925427dae2127e8f527761b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 42657 } } peers { permanent_uuid: "2774224e751644b5810fcb39969284e9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43147 } attrs { promote: false } } }
I20240503 08:08:11.173859 30626 tablet_service.cc:1514] Processing DeleteTablet for tablet 245abe56ce504129b23b513ed0f62565 with delete_type TABLET_DATA_TOMBSTONED (TS 43a08d6481084df8a886fafd6a2bfdde not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45576
I20240503 08:08:11.174160 30989 tablet_replica.cc:330] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde: stopping tablet replica
I20240503 08:08:11.174248 30989 raft_consensus.cc:2229] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:11.174316 30989 raft_consensus.cc:2258] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:11.174623 30989 ts_tablet_manager.cc:1875] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:08:11.175993 30989 ts_tablet_manager.cc:1888] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:08:11.176061 30989 log.cc:1196] T 245abe56ce504129b23b513ed0f62565 P 43a08d6481084df8a886fafd6a2bfdde: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/245abe56ce504129b23b513ed0f62565
I20240503 08:08:11.176396 30519 catalog_manager.cc:4830] TS 43a08d6481084df8a886fafd6a2bfdde (127.28.55.65:46553): tablet 245abe56ce504129b23b513ed0f62565 (table test-workload [id=9c8d26ee8f0449dc9b45e732d00e5c58]) successfully deleted
I20240503 08:08:12.736856 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:12.737021 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:12.737042 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:14.737244 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:14.737393 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:14.737414 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:16.737613 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:16.737762 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:16.737782 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:18.737973 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:18.738126 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:18.738145 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:20.738341 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:20.738502 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:20.738518 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:22.738708 30588 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:22.738909 30588 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:22.738940 30588 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:23.316615 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:08:23.319191 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:23.319322 28893 tablet_replica.cc:330] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde: stopping tablet replica
I20240503 08:08:23.319373 28893 raft_consensus.cc:2229] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:08:23.319453 28893 raft_consensus.cc:2258] T 3cff297fd92a4b218c50a228ae8ac908 P 43a08d6481084df8a886fafd6a2bfdde [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:23.331514 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:08:23.332451 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:08:23.335393 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:23.335530 28893 tablet_replica.cc:330] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b: stopping tablet replica
I20240503 08:08:23.335589 28893 raft_consensus.cc:2229] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:23.335641 28893 raft_consensus.cc:2258] T 3cff297fd92a4b218c50a228ae8ac908 P a3dc67979925427dae2127e8f527761b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:23.335822 28893 tablet_replica.cc:330] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b: stopping tablet replica
I20240503 08:08:23.335857 28893 raft_consensus.cc:2229] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:23.335893 28893 raft_consensus.cc:2258] T 245abe56ce504129b23b513ed0f62565 P a3dc67979925427dae2127e8f527761b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:23.347463 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:08:23.348479 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:08:23.351066 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:23.351222 28893 tablet_replica.cc:330] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea: stopping tablet replica
I20240503 08:08:23.351265 28893 raft_consensus.cc:2229] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:23.351306 28893 raft_consensus.cc:2258] T 3cff297fd92a4b218c50a228ae8ac908 P 6d678e76f65343b59f26d383e881aeea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:23.351451 28893 tablet_replica.cc:330] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea: stopping tablet replica
I20240503 08:08:23.351480 28893 raft_consensus.cc:2229] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:23.351517 28893 raft_consensus.cc:2258] T 245abe56ce504129b23b513ed0f62565 P 6d678e76f65343b59f26d383e881aeea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:23.363377 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:08:23.364274 28893 tablet_server.cc:178] TabletServer@127.28.55.68:0 shutting down...
I20240503 08:08:23.366541 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:23.366644 28893 tablet_replica.cc:330] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9: stopping tablet replica
I20240503 08:08:23.366710 28893 raft_consensus.cc:2229] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:23.366762 28893 raft_consensus.cc:2258] T 245abe56ce504129b23b513ed0f62565 P 2774224e751644b5810fcb39969284e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:23.378470 28893 tablet_server.cc:194] TabletServer@127.28.55.68:0 shutdown complete.
I20240503 08:08:23.379467 28893 master.cc:537] Master@127.28.55.126:44463 shutting down...
I20240503 08:08:23.381840 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:23.381917 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:23.381935 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 78c68782ba6448b7b462071e63b46452: stopping tablet replica
I20240503 08:08:23.393728 28893 master.cc:552] Master@127.28.55.126:44463 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26716 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 08:08:23.398941 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:43427
I20240503 08:08:23.399134 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:23.400300 30996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:23.400355 30994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:23.400377 30993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:23.400548 28893 server_base.cc:1034] running on GCE node
I20240503 08:08:23.400686 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:23.400713 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:23.400729 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723703400730 us; error 0 us; skew 500 ppm
I20240503 08:08:23.406430 28893 webserver.cc:460] Webserver started at http://127.28.55.126:39569/ using document root <none> and password file <none>
I20240503 08:08:23.406536 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:23.406590 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:23.406637 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:23.407164 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "a4c9b0542d134777b1e39010909f4e6b"
format_stamp: "Formatted at 2024-05-03 08:08:23 on dist-test-slave-mlgg"
I20240503 08:08:23.408053 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:23.408537 31001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:23.408660 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:23.408700 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "a4c9b0542d134777b1e39010909f4e6b"
format_stamp: "Formatted at 2024-05-03 08:08:23 on dist-test-slave-mlgg"
I20240503 08:08:23.408748 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:23.418110 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:23.418324 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:23.420903 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:43427
I20240503 08:08:23.422963 31053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:43427 every 8 connection(s)
I20240503 08:08:23.423106 31054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:23.424140 31054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b: Bootstrap starting.
I20240503 08:08:23.424429 31054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:23.424965 31054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b: No bootstrap required, opened a new log
I20240503 08:08:23.425102 31054 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c9b0542d134777b1e39010909f4e6b" member_type: VOTER }
I20240503 08:08:23.425158 31054 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:23.425175 31054 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c9b0542d134777b1e39010909f4e6b, State: Initialized, Role: FOLLOWER
I20240503 08:08:23.425222 31054 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c9b0542d134777b1e39010909f4e6b" member_type: VOTER }
I20240503 08:08:23.425261 31054 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:08:23.425280 31054 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:08:23.425304 31054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:23.425793 31054 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c9b0542d134777b1e39010909f4e6b" member_type: VOTER }
I20240503 08:08:23.425853 31054 leader_election.cc:302] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4c9b0542d134777b1e39010909f4e6b; no voters:
I20240503 08:08:23.425976 31054 leader_election.cc:288] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:08:23.426023 31057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:23.426122 31054 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:08:23.426178 31057 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 1 LEADER]: Becoming Leader. State: Replica: a4c9b0542d134777b1e39010909f4e6b, State: Running, Role: LEADER
I20240503 08:08:23.426249 31057 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c9b0542d134777b1e39010909f4e6b" member_type: VOTER }
I20240503 08:08:23.426455 31058 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4c9b0542d134777b1e39010909f4e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c9b0542d134777b1e39010909f4e6b" member_type: VOTER } }
I20240503 08:08:23.426555 31058 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:23.426466 31059 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c9b0542d134777b1e39010909f4e6b. Latest consensus state: current_term: 1 leader_uuid: "a4c9b0542d134777b1e39010909f4e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c9b0542d134777b1e39010909f4e6b" member_type: VOTER } }
I20240503 08:08:23.426869 31059 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:23.426926 31064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:08:23.427073 31064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:08:23.427425 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:08:23.427634 31064 catalog_manager.cc:1347] Generated new cluster ID: 8a3a6e76dee046a29fab16df1e9cbf7f
I20240503 08:08:23.427671 31064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:08:23.432181 31064 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:08:23.432600 31064 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:08:23.438344 31064 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b: Generated new TSK 0
I20240503 08:08:23.438447 31064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:08:23.442994 28893 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 4.2593e-05s
I20240503 08:08:35.519814 28893 master.cc:537] Master@127.28.55.126:43427 shutting down...
I20240503 08:08:35.521903 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:35.521996 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:35.522015 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P a4c9b0542d134777b1e39010909f4e6b: stopping tablet replica
I20240503 08:08:35.533516 28893 master.cc:552] Master@127.28.55.126:43427 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12137 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 08:08:35.536231 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:42899
I20240503 08:08:35.536381 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:08:35.537705 28893 server_base.cc:1034] running on GCE node
W20240503 08:08:35.537782 31076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:35.537873 31079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:35.537793 31077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:35.538048 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:35.538086 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:35.538103 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723715538103 us; error 0 us; skew 500 ppm
I20240503 08:08:35.543347 28893 webserver.cc:460] Webserver started at http://127.28.55.126:44585/ using document root <none> and password file <none>
I20240503 08:08:35.543463 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:35.543510 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:35.543557 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:35.543850 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "9a8ef0e5fd254abebbcf4925d4481527"
format_stamp: "Formatted at 2024-05-03 08:08:35 on dist-test-slave-mlgg"
I20240503 08:08:35.544745 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:35.545285 31084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.545419 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.545467 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "9a8ef0e5fd254abebbcf4925d4481527"
format_stamp: "Formatted at 2024-05-03 08:08:35 on dist-test-slave-mlgg"
I20240503 08:08:35.545511 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:35.568867 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:35.569069 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:35.571729 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:42899
I20240503 08:08:35.572242 31136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:42899 every 8 connection(s)
I20240503 08:08:35.572379 31137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:35.573493 31137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527: Bootstrap starting.
I20240503 08:08:35.573787 31137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:35.574344 31137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527: No bootstrap required, opened a new log
I20240503 08:08:35.574471 31137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8ef0e5fd254abebbcf4925d4481527" member_type: VOTER }
I20240503 08:08:35.574532 31137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:35.574548 31137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8ef0e5fd254abebbcf4925d4481527, State: Initialized, Role: FOLLOWER
I20240503 08:08:35.574599 31137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8ef0e5fd254abebbcf4925d4481527" member_type: VOTER }
I20240503 08:08:35.574630 31137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:08:35.574647 31137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:08:35.574668 31137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:35.575238 31137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8ef0e5fd254abebbcf4925d4481527" member_type: VOTER }
I20240503 08:08:35.575299 31137 leader_election.cc:302] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a8ef0e5fd254abebbcf4925d4481527; no voters:
I20240503 08:08:35.575413 31137 leader_election.cc:288] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:08:35.575490 31140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:35.575556 31137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:08:35.575634 31140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 1 LEADER]: Becoming Leader. State: Replica: 9a8ef0e5fd254abebbcf4925d4481527, State: Running, Role: LEADER
I20240503 08:08:35.575692 31140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8ef0e5fd254abebbcf4925d4481527" member_type: VOTER }
I20240503 08:08:35.575923 31141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a8ef0e5fd254abebbcf4925d4481527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8ef0e5fd254abebbcf4925d4481527" member_type: VOTER } }
I20240503 08:08:35.575951 31142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a8ef0e5fd254abebbcf4925d4481527. Latest consensus state: current_term: 1 leader_uuid: "9a8ef0e5fd254abebbcf4925d4481527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8ef0e5fd254abebbcf4925d4481527" member_type: VOTER } }
I20240503 08:08:35.576064 31142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:35.576228 31141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:35.576228 31148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:08:35.576467 31148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:08:35.576897 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:08:35.577137 31148 catalog_manager.cc:1347] Generated new cluster ID: a891b3e4dbb4486386d1acc92ced931e
I20240503 08:08:35.577181 31148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:08:35.588219 31148 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:08:35.588672 31148 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:08:35.606541 31148 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527: Generated new TSK 0
I20240503 08:08:35.606653 31148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:08:35.608758 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:35.610149 31161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:35.610214 31159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:35.610240 28893 server_base.cc:1034] running on GCE node
W20240503 08:08:35.610277 31158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:35.610419 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:35.610468 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:35.610486 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723715610485 us; error 0 us; skew 500 ppm
I20240503 08:08:35.611145 28893 webserver.cc:460] Webserver started at http://127.28.55.65:43225/ using document root <none> and password file <none>
I20240503 08:08:35.611225 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:35.611281 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:35.611332 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:35.611622 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "bba9ef35459d4a3a9b120be23fc0856b"
format_stamp: "Formatted at 2024-05-03 08:08:35 on dist-test-slave-mlgg"
I20240503 08:08:35.612506 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:35.612977 31166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.613142 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.613183 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "bba9ef35459d4a3a9b120be23fc0856b"
format_stamp: "Formatted at 2024-05-03 08:08:35 on dist-test-slave-mlgg"
I20240503 08:08:35.613231 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:35.634934 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:35.635172 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:35.635344 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:35.635558 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:35.635588 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.635617 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:35.635644 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.638330 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:37427
I20240503 08:08:35.638391 31229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:37427 every 8 connection(s)
I20240503 08:08:35.638707 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:35.639991 31234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:35.640062 31237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:35.640129 28893 server_base.cc:1034] running on GCE node
W20240503 08:08:35.640492 31235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:35.640661 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:35.640697 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:35.640712 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723715640712 us; error 0 us; skew 500 ppm
I20240503 08:08:35.641499 28893 webserver.cc:460] Webserver started at http://127.28.55.66:33127/ using document root <none> and password file <none>
I20240503 08:08:35.641592 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:35.641645 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:35.641695 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:35.641976 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "2f0b59ebd5be489ab891229c2e0d0d9d"
format_stamp: "Formatted at 2024-05-03 08:08:35 on dist-test-slave-mlgg"
I20240503 08:08:35.642323 31230 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42899
I20240503 08:08:35.642385 31230 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:35.642489 31230 heartbeater.cc:511] Master 127.28.55.126:42899 requested a full tablet report, sending...
I20240503 08:08:35.642709 31101 ts_manager.cc:194] Registered new tserver with Master: bba9ef35459d4a3a9b120be23fc0856b (127.28.55.65:37427)
I20240503 08:08:35.643015 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:35.643287 31101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57394
I20240503 08:08:35.643479 31242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.643605 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.643658 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "2f0b59ebd5be489ab891229c2e0d0d9d"
format_stamp: "Formatted at 2024-05-03 08:08:35 on dist-test-slave-mlgg"
I20240503 08:08:35.643704 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:35.655316 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:35.655510 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:35.655665 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:35.655851 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:35.655880 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.655917 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:35.655941 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.658710 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:43603
I20240503 08:08:35.658766 31305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:43603 every 8 connection(s)
I20240503 08:08:35.659103 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:35.660286 31309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:35.660324 31312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:35.660409 28893 server_base.cc:1034] running on GCE node
W20240503 08:08:35.660759 31310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:35.660933 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:35.660969 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:35.660986 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723715660986 us; error 0 us; skew 500 ppm
I20240503 08:08:35.661717 28893 webserver.cc:460] Webserver started at http://127.28.55.67:42603/ using document root <none> and password file <none>
I20240503 08:08:35.661820 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:35.661863 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:35.661903 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:35.662222 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "052301359f5d4a6bbdda647c8e70ac6a"
format_stamp: "Formatted at 2024-05-03 08:08:35 on dist-test-slave-mlgg"
I20240503 08:08:35.662281 31306 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42899
I20240503 08:08:35.662356 31306 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:35.662467 31306 heartbeater.cc:511] Master 127.28.55.126:42899 requested a full tablet report, sending...
I20240503 08:08:35.662694 31101 ts_manager.cc:194] Registered new tserver with Master: 2f0b59ebd5be489ab891229c2e0d0d9d (127.28.55.66:43603)
I20240503 08:08:35.663086 31101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57404
I20240503 08:08:35.663348 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:35.663815 31317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.663944 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.663987 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "052301359f5d4a6bbdda647c8e70ac6a"
format_stamp: "Formatted at 2024-05-03 08:08:35 on dist-test-slave-mlgg"
I20240503 08:08:35.664033 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:35.671519 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:35.671741 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:35.671913 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:35.672091 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:35.672117 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.672144 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:35.672168 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:35.675278 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:39659
I20240503 08:08:35.675355 31380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:39659 every 8 connection(s)
I20240503 08:08:35.678329 31381 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42899
I20240503 08:08:35.678392 31381 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:35.678476 31381 heartbeater.cc:511] Master 127.28.55.126:42899 requested a full tablet report, sending...
I20240503 08:08:35.678692 31101 ts_manager.cc:194] Registered new tserver with Master: 052301359f5d4a6bbdda647c8e70ac6a (127.28.55.67:39659)
I20240503 08:08:35.678798 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003357544s
I20240503 08:08:35.679173 31101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57418
I20240503 08:08:36.643841 31230 heartbeater.cc:503] Master 127.28.55.126:42899 was elected leader, sending a full tablet report...
I20240503 08:08:36.663635 31306 heartbeater.cc:503] Master 127.28.55.126:42899 was elected leader, sending a full tablet report...
I20240503 08:08:36.679675 31381 heartbeater.cc:503] Master 127.28.55.126:42899 was elected leader, sending a full tablet report...
I20240503 08:08:37.576877 31155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:39.577047 31155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:41.577299 31155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:43.577544 31155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:45.577780 31155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:47.577999 31155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:47.755383 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:08:47.757890 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:47.769487 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:08:47.770424 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:08:47.773069 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:47.784678 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:08:47.785506 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:08:47.787995 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:47.799489 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:08:47.800330 28893 master.cc:537] Master@127.28.55.126:42899 shutting down...
I20240503 08:08:47.802700 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:47.802785 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:47.802821 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9a8ef0e5fd254abebbcf4925d4481527: stopping tablet replica
I20240503 08:08:47.814419 28893 master.cc:552] Master@127.28.55.126:42899 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12281 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 08:08:47.817863 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:36683
I20240503 08:08:47.818011 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:47.819547 31385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.819550 31388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.819664 31386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.819584 28893 server_base.cc:1034] running on GCE node
I20240503 08:08:47.819809 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:47.819943 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:47.819955 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723727819955 us; error 0 us; skew 500 ppm
I20240503 08:08:47.825767 28893 webserver.cc:460] Webserver started at http://127.28.55.126:41819/ using document root <none> and password file <none>
I20240503 08:08:47.825865 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:47.825901 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:47.825937 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:47.826202 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "c9d5faf536ee41088871b92075d30358"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.827085 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:47.827584 31393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.827708 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.827752 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "c9d5faf536ee41088871b92075d30358"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.827800 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:47.840904 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:47.841112 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:47.843675 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:36683
I20240503 08:08:47.844830 31445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:36683 every 8 connection(s)
I20240503 08:08:47.844966 31446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:47.846029 31446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358: Bootstrap starting.
I20240503 08:08:47.846274 31446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:47.846827 31446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358: No bootstrap required, opened a new log
I20240503 08:08:47.846946 31446 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5faf536ee41088871b92075d30358" member_type: VOTER }
I20240503 08:08:47.846997 31446 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:47.847007 31446 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d5faf536ee41088871b92075d30358, State: Initialized, Role: FOLLOWER
I20240503 08:08:47.847043 31446 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5faf536ee41088871b92075d30358" member_type: VOTER }
I20240503 08:08:47.847079 31446 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:08:47.847097 31446 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:08:47.847121 31446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:47.847604 31446 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5faf536ee41088871b92075d30358" member_type: VOTER }
I20240503 08:08:47.847671 31446 leader_election.cc:302] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d5faf536ee41088871b92075d30358; no voters:
I20240503 08:08:47.847769 31446 leader_election.cc:288] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:08:47.847841 31449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:47.847942 31446 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:08:47.847986 31449 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 1 LEADER]: Becoming Leader. State: Replica: c9d5faf536ee41088871b92075d30358, State: Running, Role: LEADER
I20240503 08:08:47.848074 31449 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5faf536ee41088871b92075d30358" member_type: VOTER }
I20240503 08:08:47.848291 31450 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d5faf536ee41088871b92075d30358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5faf536ee41088871b92075d30358" member_type: VOTER } }
I20240503 08:08:47.848292 31451 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d5faf536ee41088871b92075d30358. Latest consensus state: current_term: 1 leader_uuid: "c9d5faf536ee41088871b92075d30358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5faf536ee41088871b92075d30358" member_type: VOTER } }
I20240503 08:08:47.848413 31451 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:47.848575 31450 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:47.848683 31455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:08:47.848815 31455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:08:47.849166 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:08:47.849386 31455 catalog_manager.cc:1347] Generated new cluster ID: 06d3567cf4fa4cf98a1dafc17e4f00f9
I20240503 08:08:47.849436 31455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:08:47.855152 31455 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:08:47.855573 31455 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:08:47.860968 31455 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358: Generated new TSK 0
I20240503 08:08:47.861047 31455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:08:47.864928 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:47.866205 31467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.866325 28893 server_base.cc:1034] running on GCE node
W20240503 08:08:47.866289 31470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.866263 31468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.866565 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:47.866598 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:47.866613 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723727866614 us; error 0 us; skew 500 ppm
I20240503 08:08:47.867276 28893 webserver.cc:460] Webserver started at http://127.28.55.65:33353/ using document root <none> and password file <none>
I20240503 08:08:47.867364 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:47.867409 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:47.867461 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:47.867733 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "5426c733ac4d47b2b4a285b7e07c5624"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.868580 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:47.868978 31475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.869107 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.869144 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "5426c733ac4d47b2b4a285b7e07c5624"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.869191 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:47.883567 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:47.883801 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:47.883960 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:47.884159 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:47.884187 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.884215 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:47.884239 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.886970 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:40883
I20240503 08:08:47.887007 31538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:40883 every 8 connection(s)
I20240503 08:08:47.887398 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:47.888625 31546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.888733 28893 server_base.cc:1034] running on GCE node
W20240503 08:08:47.888645 31543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.888868 31544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.889037 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:47.889076 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:47.889092 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723727889093 us; error 0 us; skew 500 ppm
I20240503 08:08:47.889910 28893 webserver.cc:460] Webserver started at http://127.28.55.66:46117/ using document root <none> and password file <none>
I20240503 08:08:47.890025 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:47.890071 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:47.890121 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:47.890507 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "0eb796328eb9425c9bdad4fbbf6fbbb8"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.891244 31539 heartbeater.cc:347] Connected to a master server at 127.28.55.126:36683
I20240503 08:08:47.891311 31539 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:47.891423 31539 heartbeater.cc:511] Master 127.28.55.126:36683 requested a full tablet report, sending...
I20240503 08:08:47.891623 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:08:47.891719 31410 ts_manager.cc:194] Registered new tserver with Master: 5426c733ac4d47b2b4a285b7e07c5624 (127.28.55.65:40883)
I20240503 08:08:47.892174 31551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.892285 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:08:47.892329 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "0eb796328eb9425c9bdad4fbbf6fbbb8"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.892380 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:47.892429 31410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45734
I20240503 08:08:47.901110 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:47.901345 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:47.901510 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:47.901712 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:47.901738 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.901767 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:47.901789 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.904779 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:42059
I20240503 08:08:47.904816 31614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:42059 every 8 connection(s)
I20240503 08:08:47.905194 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:47.906479 31618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.906579 31619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.906507 31621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.906951 28893 server_base.cc:1034] running on GCE node
I20240503 08:08:47.907063 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:47.907094 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:47.907109 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723727907110 us; error 0 us; skew 500 ppm
I20240503 08:08:47.907855 28893 webserver.cc:460] Webserver started at http://127.28.55.67:43975/ using document root <none> and password file <none>
I20240503 08:08:47.907959 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:47.908007 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:47.908051 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:47.908303 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "92b1cc778f124b7da82e9cc07c8ad2ba"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.908733 31615 heartbeater.cc:347] Connected to a master server at 127.28.55.126:36683
I20240503 08:08:47.908789 31615 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:47.908870 31615 heartbeater.cc:511] Master 127.28.55.126:36683 requested a full tablet report, sending...
I20240503 08:08:47.909116 31410 ts_manager.cc:194] Registered new tserver with Master: 0eb796328eb9425c9bdad4fbbf6fbbb8 (127.28.55.66:42059)
I20240503 08:08:47.909245 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:08:47.909495 31410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45740
I20240503 08:08:47.909716 31626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.909842 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.909894 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "92b1cc778f124b7da82e9cc07c8ad2ba"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.909945 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:47.931560 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:47.931840 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:47.932021 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:47.932317 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:47.932354 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.932384 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:47.932407 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.936419 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:37147
I20240503 08:08:47.936481 31689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:37147 every 8 connection(s)
I20240503 08:08:47.936851 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:47.938258 31693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.938334 28893 server_base.cc:1034] running on GCE node
W20240503 08:08:47.940110 31694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.940361 31696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.940517 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:47.940557 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:47.940568 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723727940569 us; error 0 us; skew 500 ppm
I20240503 08:08:47.941334 31690 heartbeater.cc:347] Connected to a master server at 127.28.55.126:36683
I20240503 08:08:47.941409 31690 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:47.941416 28893 webserver.cc:460] Webserver started at http://127.28.55.68:40325/ using document root <none> and password file <none>
I20240503 08:08:47.941535 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:47.941555 31690 heartbeater.cc:511] Master 127.28.55.126:36683 requested a full tablet report, sending...
I20240503 08:08:47.941578 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:47.941615 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:47.941790 31410 ts_manager.cc:194] Registered new tserver with Master: 92b1cc778f124b7da82e9cc07c8ad2ba (127.28.55.67:37147)
I20240503 08:08:47.941982 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-3-root/instance:
uuid: "df96411708444364b5b14f47aa5850fb"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.942246 31410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45750
I20240503 08:08:47.942981 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:47.943467 31701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.943836 28893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:08:47.943881 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-3-root
uuid: "df96411708444364b5b14f47aa5850fb"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-mlgg"
I20240503 08:08:47.943933 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714723660874102-28893-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:47.959266 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:47.959571 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:47.959748 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:47.959959 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:47.959987 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.960016 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:47.960040 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.963776 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.68:42057
I20240503 08:08:47.964005 31764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.68:42057 every 8 connection(s)
I20240503 08:08:47.967543 31765 heartbeater.cc:347] Connected to a master server at 127.28.55.126:36683
I20240503 08:08:47.967626 31765 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:47.967746 31765 heartbeater.cc:511] Master 127.28.55.126:36683 requested a full tablet report, sending...
I20240503 08:08:47.967979 31410 ts_manager.cc:194] Registered new tserver with Master: df96411708444364b5b14f47aa5850fb (127.28.55.68:42057)
I20240503 08:08:47.968369 28893 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.004420348s
I20240503 08:08:47.968370 31410 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45766
I20240503 08:08:48.892971 31539 heartbeater.cc:503] Master 127.28.55.126:36683 was elected leader, sending a full tablet report...
I20240503 08:08:48.910048 31615 heartbeater.cc:503] Master 127.28.55.126:36683 was elected leader, sending a full tablet report...
I20240503 08:08:48.942775 31690 heartbeater.cc:503] Master 127.28.55.126:36683 was elected leader, sending a full tablet report...
I20240503 08:08:48.969130 31765 heartbeater.cc:503] Master 127.28.55.126:36683 was elected leader, sending a full tablet report...
I20240503 08:08:48.996984 28893 test_util.cc:268] Using random seed: 985741940
I20240503 08:08:49.001365 31410 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45786:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 08:08:49.006466 31729 tablet_service.cc:1467] Processing CreateTablet for tablet e3d9d79b5a264cdc88c1c01aa58ed240 (DEFAULT_TABLE table=test-workload [id=89005146f61d441aa1f8f2f861df64e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:49.006522 31503 tablet_service.cc:1467] Processing CreateTablet for tablet e3d9d79b5a264cdc88c1c01aa58ed240 (DEFAULT_TABLE table=test-workload [id=89005146f61d441aa1f8f2f861df64e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:49.006597 31654 tablet_service.cc:1467] Processing CreateTablet for tablet e3d9d79b5a264cdc88c1c01aa58ed240 (DEFAULT_TABLE table=test-workload [id=89005146f61d441aa1f8f2f861df64e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:49.006635 31729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d9d79b5a264cdc88c1c01aa58ed240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:49.006675 31654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d9d79b5a264cdc88c1c01aa58ed240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:49.006645 31503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d9d79b5a264cdc88c1c01aa58ed240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:49.008047 31786 tablet_bootstrap.cc:492] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb: Bootstrap starting.
I20240503 08:08:49.008368 31787 tablet_bootstrap.cc:492] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba: Bootstrap starting.
I20240503 08:08:49.008436 31786 tablet_bootstrap.cc:654] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:49.008772 31787 tablet_bootstrap.cc:654] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:49.009037 31786 tablet_bootstrap.cc:492] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb: No bootstrap required, opened a new log
I20240503 08:08:49.009084 31788 tablet_bootstrap.cc:492] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624: Bootstrap starting.
I20240503 08:08:49.009094 31786 ts_tablet_manager.cc:1367] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:49.009305 31787 tablet_bootstrap.cc:492] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba: No bootstrap required, opened a new log
I20240503 08:08:49.009357 31787 ts_tablet_manager.cc:1367] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:49.009344 31786 raft_consensus.cc:348] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } }
I20240503 08:08:49.009455 31786 raft_consensus.cc:374] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:49.009496 31786 raft_consensus.cc:729] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df96411708444364b5b14f47aa5850fb, State: Initialized, Role: FOLLOWER
I20240503 08:08:49.009565 31788 tablet_bootstrap.cc:654] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:49.009588 31787 raft_consensus.cc:348] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } }
I20240503 08:08:49.009552 31786 consensus_queue.cc:260] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } }
I20240503 08:08:49.009666 31787 raft_consensus.cc:374] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:49.009752 31787 raft_consensus.cc:729] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b1cc778f124b7da82e9cc07c8ad2ba, State: Initialized, Role: FOLLOWER
I20240503 08:08:49.009814 31787 consensus_queue.cc:260] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } }
I20240503 08:08:49.009891 31786 ts_tablet_manager.cc:1398] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:49.009970 31787 ts_tablet_manager.cc:1398] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:08:49.010219 31788 tablet_bootstrap.cc:492] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624: No bootstrap required, opened a new log
I20240503 08:08:49.010308 31788 ts_tablet_manager.cc:1367] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:49.010514 31788 raft_consensus.cc:348] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } }
I20240503 08:08:49.010579 31788 raft_consensus.cc:374] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:49.010598 31788 raft_consensus.cc:729] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5426c733ac4d47b2b4a285b7e07c5624, State: Initialized, Role: FOLLOWER
I20240503 08:08:49.010637 31788 consensus_queue.cc:260] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } }
I20240503 08:08:49.010790 31788 ts_tablet_manager.cc:1398] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:49.019840 31791 raft_consensus.cc:482] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:49.019937 31791 raft_consensus.cc:504] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } }
I20240503 08:08:49.020172 31791 leader_election.cc:288] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b1cc778f124b7da82e9cc07c8ad2ba (127.28.55.67:37147), 5426c733ac4d47b2b4a285b7e07c5624 (127.28.55.65:40883)
I20240503 08:08:49.022747 31513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d9d79b5a264cdc88c1c01aa58ed240" candidate_uuid: "df96411708444364b5b14f47aa5850fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5426c733ac4d47b2b4a285b7e07c5624" is_pre_election: true
I20240503 08:08:49.022758 31664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d9d79b5a264cdc88c1c01aa58ed240" candidate_uuid: "df96411708444364b5b14f47aa5850fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" is_pre_election: true
I20240503 08:08:49.022907 31513 raft_consensus.cc:2454] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df96411708444364b5b14f47aa5850fb in term 0.
I20240503 08:08:49.022909 31664 raft_consensus.cc:2454] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df96411708444364b5b14f47aa5850fb in term 0.
I20240503 08:08:49.023061 31702 leader_election.cc:302] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5426c733ac4d47b2b4a285b7e07c5624, df96411708444364b5b14f47aa5850fb; no voters:
I20240503 08:08:49.023159 31791 raft_consensus.cc:2781] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:08:49.023202 31791 raft_consensus.cc:482] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:08:49.023216 31791 raft_consensus.cc:3037] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:49.023859 31791 raft_consensus.cc:504] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } }
I20240503 08:08:49.023978 31791 leader_election.cc:288] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [CANDIDATE]: Term 1 election: Requested vote from peers 92b1cc778f124b7da82e9cc07c8ad2ba (127.28.55.67:37147), 5426c733ac4d47b2b4a285b7e07c5624 (127.28.55.65:40883)
I20240503 08:08:49.024119 31513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d9d79b5a264cdc88c1c01aa58ed240" candidate_uuid: "df96411708444364b5b14f47aa5850fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5426c733ac4d47b2b4a285b7e07c5624"
I20240503 08:08:49.024120 31664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d9d79b5a264cdc88c1c01aa58ed240" candidate_uuid: "df96411708444364b5b14f47aa5850fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba"
I20240503 08:08:49.024194 31513 raft_consensus.cc:3037] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:49.024210 31664 raft_consensus.cc:3037] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:49.024945 31664 raft_consensus.cc:2454] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df96411708444364b5b14f47aa5850fb in term 1.
I20240503 08:08:49.024960 31513 raft_consensus.cc:2454] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df96411708444364b5b14f47aa5850fb in term 1.
I20240503 08:08:49.025121 31703 leader_election.cc:302] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92b1cc778f124b7da82e9cc07c8ad2ba, df96411708444364b5b14f47aa5850fb; no voters:
I20240503 08:08:49.025206 31791 raft_consensus.cc:2781] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:49.025355 31791 raft_consensus.cc:686] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 1 LEADER]: Becoming Leader. State: Replica: df96411708444364b5b14f47aa5850fb, State: Running, Role: LEADER
I20240503 08:08:49.025404 31791 consensus_queue.cc:237] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } }
I20240503 08:08:49.025959 31409 catalog_manager.cc:5484] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb reported cstate change: term changed from 0 to 1, leader changed from <none> to df96411708444364b5b14f47aa5850fb (127.28.55.68). New cstate: current_term: 1 leader_uuid: "df96411708444364b5b14f47aa5850fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df96411708444364b5b14f47aa5850fb" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 42057 } health_report { overall_health: HEALTHY } } }
W20240503 08:08:49.138067 31540 tablet.cc:2369] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:08:49.445473 31791 consensus_queue.cc:1035] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b1cc778f124b7da82e9cc07c8ad2ba" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:49.447258 31802 consensus_queue.cc:1035] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5426c733ac4d47b2b4a285b7e07c5624" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 40883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:49.849288 31464 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:49.849421 31464 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:49.849449 31464 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:51.849635 31464 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:51.849800 31464 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:51.849822 31464 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:53.849983 31464 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:53.850147 31464 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:53.850168 31464 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:55.850334 31464 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:55.850461 31464 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:55.850486 31464 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:57.850643 31464 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:57.850792 31464 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:57.850857 31464 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:08:59.851028 31464 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:08:59.851187 31464 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:08:59.851219 31464 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:00.082141 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:09:00.084600 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:00.084757 28893 tablet_replica.cc:330] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624: stopping tablet replica
I20240503 08:09:00.084815 28893 raft_consensus.cc:2229] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:00.084868 28893 raft_consensus.cc:2258] T e3d9d79b5a264cdc88c1c01aa58ed240 P 5426c733ac4d47b2b4a285b7e07c5624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:00.098291 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:09:00.099308 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:09:00.101975 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:00.113276 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:09:00.114092 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:09:00.116304 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:00.116443 28893 tablet_replica.cc:330] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba: stopping tablet replica
I20240503 08:09:00.116503 28893 raft_consensus.cc:2229] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:00.116542 28893 raft_consensus.cc:2258] T e3d9d79b5a264cdc88c1c01aa58ed240 P 92b1cc778f124b7da82e9cc07c8ad2ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:00.128106 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:09:00.129026 28893 tablet_server.cc:178] TabletServer@127.28.55.68:0 shutting down...
I20240503 08:09:00.131296 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:00.131419 28893 tablet_replica.cc:330] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb: stopping tablet replica
I20240503 08:09:00.131474 28893 raft_consensus.cc:2229] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:00.131548 28893 raft_consensus.cc:2258] T e3d9d79b5a264cdc88c1c01aa58ed240 P df96411708444364b5b14f47aa5850fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:00.143378 28893 tablet_server.cc:194] TabletServer@127.28.55.68:0 shutdown complete.
I20240503 08:09:00.144374 28893 master.cc:537] Master@127.28.55.126:36683 shutting down...
I20240503 08:09:00.146602 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:00.146701 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:00.146720 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P c9d5faf536ee41088871b92075d30358: stopping tablet replica
I20240503 08:09:00.158521 28893 master.cc:552] Master@127.28.55.126:36683 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12345 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 08:09:00.163089 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:41999
I20240503 08:09:00.163266 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:00.164660 31808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:00.164680 31807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:00.164716 31810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:00.164764 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:00.165017 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:00.165052 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:00.165069 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723740165069 us; error 0 us; skew 500 ppm
I20240503 08:09:00.165700 28893 webserver.cc:460] Webserver started at http://127.28.55.126:42193/ using document root <none> and password file <none>
I20240503 08:09:00.165777 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:00.165827 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:00.165879 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:00.166189 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "25e4f4497aa74b3b83fe172f4434fc69"
format_stamp: "Formatted at 2024-05-03 08:09:00 on dist-test-slave-mlgg"
I20240503 08:09:00.167196 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:00.167659 31815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.167786 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.167827 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "25e4f4497aa74b3b83fe172f4434fc69"
format_stamp: "Formatted at 2024-05-03 08:09:00 on dist-test-slave-mlgg"
I20240503 08:09:00.167881 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:00.172616 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:00.172797 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:00.175221 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:41999
I20240503 08:09:00.176654 31867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:41999 every 8 connection(s)
I20240503 08:09:00.176787 31868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:00.177805 31868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69: Bootstrap starting.
I20240503 08:09:00.178094 31868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:00.178606 31868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69: No bootstrap required, opened a new log
I20240503 08:09:00.178726 31868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e4f4497aa74b3b83fe172f4434fc69" member_type: VOTER }
I20240503 08:09:00.178781 31868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:00.178798 31868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e4f4497aa74b3b83fe172f4434fc69, State: Initialized, Role: FOLLOWER
I20240503 08:09:00.178864 31868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e4f4497aa74b3b83fe172f4434fc69" member_type: VOTER }
I20240503 08:09:00.178910 31868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:09:00.178928 31868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:09:00.178954 31868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:00.179498 31868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e4f4497aa74b3b83fe172f4434fc69" member_type: VOTER }
I20240503 08:09:00.179562 31868 leader_election.cc:302] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25e4f4497aa74b3b83fe172f4434fc69; no voters:
I20240503 08:09:00.179672 31868 leader_election.cc:288] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:09:00.179705 31871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:00.179811 31868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:09:00.179885 31871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 1 LEADER]: Becoming Leader. State: Replica: 25e4f4497aa74b3b83fe172f4434fc69, State: Running, Role: LEADER
I20240503 08:09:00.179945 31871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e4f4497aa74b3b83fe172f4434fc69" member_type: VOTER }
I20240503 08:09:00.180143 31872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25e4f4497aa74b3b83fe172f4434fc69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e4f4497aa74b3b83fe172f4434fc69" member_type: VOTER } }
I20240503 08:09:00.180234 31872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:00.180150 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e4f4497aa74b3b83fe172f4434fc69. Latest consensus state: current_term: 1 leader_uuid: "25e4f4497aa74b3b83fe172f4434fc69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e4f4497aa74b3b83fe172f4434fc69" member_type: VOTER } }
I20240503 08:09:00.180269 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:00.180542 31877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:09:00.180666 31877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:09:00.181018 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:09:00.181197 31877 catalog_manager.cc:1347] Generated new cluster ID: dd966e78628f4c6da1bdada5bc918a48
I20240503 08:09:00.181243 31877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:09:00.187686 31877 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:09:00.188087 31877 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:09:00.200692 31877 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69: Generated new TSK 0
I20240503 08:09:00.200804 31877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:09:00.212895 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:00.214293 31889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:00.214375 31892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:00.214426 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:00.214380 31890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:00.214641 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:00.214704 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:00.214720 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723740214720 us; error 0 us; skew 500 ppm
I20240503 08:09:00.215458 28893 webserver.cc:460] Webserver started at http://127.28.55.65:45287/ using document root <none> and password file <none>
I20240503 08:09:00.215543 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:00.215596 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:00.215643 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:00.215942 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "e897257feba3422a9806b63ceabfc4c6"
format_stamp: "Formatted at 2024-05-03 08:09:00 on dist-test-slave-mlgg"
I20240503 08:09:00.216850 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:00.217327 31897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.217448 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:09:00.217491 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "e897257feba3422a9806b63ceabfc4c6"
format_stamp: "Formatted at 2024-05-03 08:09:00 on dist-test-slave-mlgg"
I20240503 08:09:00.217540 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:00.227805 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:00.228003 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:00.228166 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:00.228359 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:00.228386 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.228413 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:00.228439 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.231145 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:33781
I20240503 08:09:00.231176 31960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:33781 every 8 connection(s)
I20240503 08:09:00.231529 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:00.232678 31966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:00.232751 31965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:00.232684 31968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:00.232913 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:00.233052 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:00.233085 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:00.233111 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723740233111 us; error 0 us; skew 500 ppm
I20240503 08:09:00.233778 28893 webserver.cc:460] Webserver started at http://127.28.55.66:33565/ using document root <none> and password file <none>
I20240503 08:09:00.233865 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:00.233916 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:00.233963 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:00.234283 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "aa7dfbe8a7334da8944c044ad1b42860"
format_stamp: "Formatted at 2024-05-03 08:09:00 on dist-test-slave-mlgg"
I20240503 08:09:00.235338 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:00.235355 31961 heartbeater.cc:347] Connected to a master server at 127.28.55.126:41999
I20240503 08:09:00.235409 31961 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:00.235525 31961 heartbeater.cc:511] Master 127.28.55.126:41999 requested a full tablet report, sending...
I20240503 08:09:00.235774 31832 ts_manager.cc:194] Registered new tserver with Master: e897257feba3422a9806b63ceabfc4c6 (127.28.55.65:33781)
I20240503 08:09:00.235842 31973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.235973 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.236016 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "aa7dfbe8a7334da8944c044ad1b42860"
format_stamp: "Formatted at 2024-05-03 08:09:00 on dist-test-slave-mlgg"
I20240503 08:09:00.236066 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:00.236439 31832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57458
I20240503 08:09:00.275066 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:00.275311 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:00.275482 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:00.275701 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:00.275729 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.275758 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:00.275779 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.278460 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:36237
I20240503 08:09:00.278537 32036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:36237 every 8 connection(s)
I20240503 08:09:00.278851 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:00.280057 32040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:00.280119 32043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:00.280315 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:00.280877 32041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:00.281080 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:00.281131 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:00.281147 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723740281148 us; error 0 us; skew 500 ppm
I20240503 08:09:00.281917 28893 webserver.cc:460] Webserver started at http://127.28.55.67:43401/ using document root <none> and password file <none>
I20240503 08:09:00.282011 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:00.282084 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:00.282135 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:00.282442 32037 heartbeater.cc:347] Connected to a master server at 127.28.55.126:41999
I20240503 08:09:00.282502 32037 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:00.282495 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "19ae25ffaa8a43ebb69a3339ddc32189"
format_stamp: "Formatted at 2024-05-03 08:09:00 on dist-test-slave-mlgg"
I20240503 08:09:00.282653 32037 heartbeater.cc:511] Master 127.28.55.126:41999 requested a full tablet report, sending...
I20240503 08:09:00.282913 31832 ts_manager.cc:194] Registered new tserver with Master: aa7dfbe8a7334da8944c044ad1b42860 (127.28.55.66:36237)
I20240503 08:09:00.283265 31832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57474
I20240503 08:09:00.283505 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:00.284003 32048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.284134 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.284178 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "19ae25ffaa8a43ebb69a3339ddc32189"
format_stamp: "Formatted at 2024-05-03 08:09:00 on dist-test-slave-mlgg"
I20240503 08:09:00.284220 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:00.288472 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:00.288645 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:00.288786 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:00.288965 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:00.288992 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.289017 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:00.289037 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:00.292119 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:33287
I20240503 08:09:00.292191 32111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:33287 every 8 connection(s)
I20240503 08:09:00.295015 32112 heartbeater.cc:347] Connected to a master server at 127.28.55.126:41999
I20240503 08:09:00.295073 32112 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:00.295190 32112 heartbeater.cc:511] Master 127.28.55.126:41999 requested a full tablet report, sending...
I20240503 08:09:00.295480 31832 ts_manager.cc:194] Registered new tserver with Master: 19ae25ffaa8a43ebb69a3339ddc32189 (127.28.55.67:33287)
I20240503 08:09:00.295615 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003307902s
I20240503 08:09:00.295850 31832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57486
I20240503 08:09:01.237043 31961 heartbeater.cc:503] Master 127.28.55.126:41999 was elected leader, sending a full tablet report...
I20240503 08:09:01.283826 32037 heartbeater.cc:503] Master 127.28.55.126:41999 was elected leader, sending a full tablet report...
I20240503 08:09:01.296357 32112 heartbeater.cc:503] Master 127.28.55.126:41999 was elected leader, sending a full tablet report...
I20240503 08:09:01.324033 28893 test_util.cc:268] Using random seed: 998068989
I20240503 08:09:01.329033 31832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57500:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:09:01.329188 31832 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:09:01.334089 32001 tablet_service.cc:1467] Processing CreateTablet for tablet 50005ae686904e35b777d1375074a45e (DEFAULT_TABLE table=test-workload [id=0083ec6c71394a0ab3c168bf091eface]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:01.334259 32001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50005ae686904e35b777d1375074a45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:01.334303 31925 tablet_service.cc:1467] Processing CreateTablet for tablet 50005ae686904e35b777d1375074a45e (DEFAULT_TABLE table=test-workload [id=0083ec6c71394a0ab3c168bf091eface]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:01.334378 31925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50005ae686904e35b777d1375074a45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:01.335415 32076 tablet_service.cc:1467] Processing CreateTablet for tablet 50005ae686904e35b777d1375074a45e (DEFAULT_TABLE table=test-workload [id=0083ec6c71394a0ab3c168bf091eface]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:01.335512 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50005ae686904e35b777d1375074a45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:01.335853 32134 tablet_bootstrap.cc:492] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860: Bootstrap starting.
I20240503 08:09:01.336320 32134 tablet_bootstrap.cc:654] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:01.336722 32133 tablet_bootstrap.cc:492] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6: Bootstrap starting.
I20240503 08:09:01.336733 32136 tablet_bootstrap.cc:492] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189: Bootstrap starting.
I20240503 08:09:01.336961 32134 tablet_bootstrap.cc:492] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860: No bootstrap required, opened a new log
I20240503 08:09:01.337036 32134 ts_tablet_manager.cc:1367] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:01.337172 32133 tablet_bootstrap.cc:654] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:01.337249 32136 tablet_bootstrap.cc:654] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:01.337200 32134 raft_consensus.cc:348] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } }
I20240503 08:09:01.337355 32134 raft_consensus.cc:374] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:01.337406 32134 raft_consensus.cc:729] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7dfbe8a7334da8944c044ad1b42860, State: Initialized, Role: FOLLOWER
I20240503 08:09:01.337477 32134 consensus_queue.cc:260] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } }
I20240503 08:09:01.337652 32134 ts_tablet_manager.cc:1398] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:01.337713 32133 tablet_bootstrap.cc:492] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6: No bootstrap required, opened a new log
I20240503 08:09:01.337771 32133 ts_tablet_manager.cc:1367] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:01.337929 32133 raft_consensus.cc:348] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } }
I20240503 08:09:01.337987 32133 raft_consensus.cc:374] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:01.338001 32133 raft_consensus.cc:729] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e897257feba3422a9806b63ceabfc4c6, State: Initialized, Role: FOLLOWER
I20240503 08:09:01.338035 32133 consensus_queue.cc:260] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } }
I20240503 08:09:01.338222 32133 ts_tablet_manager.cc:1398] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:09:01.338589 32136 tablet_bootstrap.cc:492] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189: No bootstrap required, opened a new log
I20240503 08:09:01.338630 32136 ts_tablet_manager.cc:1367] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:01.338837 32136 raft_consensus.cc:348] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } }
I20240503 08:09:01.338912 32136 raft_consensus.cc:374] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:01.338932 32136 raft_consensus.cc:729] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ae25ffaa8a43ebb69a3339ddc32189, State: Initialized, Role: FOLLOWER
I20240503 08:09:01.338991 32136 consensus_queue.cc:260] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } }
I20240503 08:09:01.339149 32136 ts_tablet_manager.cc:1398] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:01.490054 32139 raft_consensus.cc:482] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:01.490192 32139 raft_consensus.cc:504] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } }
I20240503 08:09:01.490376 32139 leader_election.cc:288] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19ae25ffaa8a43ebb69a3339ddc32189 (127.28.55.67:33287), e897257feba3422a9806b63ceabfc4c6 (127.28.55.65:33781)
I20240503 08:09:01.493316 32086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50005ae686904e35b777d1375074a45e" candidate_uuid: "aa7dfbe8a7334da8944c044ad1b42860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" is_pre_election: true
I20240503 08:09:01.493315 31935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50005ae686904e35b777d1375074a45e" candidate_uuid: "aa7dfbe8a7334da8944c044ad1b42860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e897257feba3422a9806b63ceabfc4c6" is_pre_election: true
I20240503 08:09:01.493474 32086 raft_consensus.cc:2454] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7dfbe8a7334da8944c044ad1b42860 in term 0.
I20240503 08:09:01.493487 31935 raft_consensus.cc:2454] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7dfbe8a7334da8944c044ad1b42860 in term 0.
I20240503 08:09:01.493639 31976 leader_election.cc:302] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa7dfbe8a7334da8944c044ad1b42860, e897257feba3422a9806b63ceabfc4c6; no voters:
I20240503 08:09:01.493724 32139 raft_consensus.cc:2781] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:01.493767 32139 raft_consensus.cc:482] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:01.493785 32139 raft_consensus.cc:3037] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:01.494431 32139 raft_consensus.cc:504] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } }
I20240503 08:09:01.494553 32139 leader_election.cc:288] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [CANDIDATE]: Term 1 election: Requested vote from peers 19ae25ffaa8a43ebb69a3339ddc32189 (127.28.55.67:33287), e897257feba3422a9806b63ceabfc4c6 (127.28.55.65:33781)
I20240503 08:09:01.494729 32086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50005ae686904e35b777d1375074a45e" candidate_uuid: "aa7dfbe8a7334da8944c044ad1b42860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ae25ffaa8a43ebb69a3339ddc32189"
I20240503 08:09:01.494750 31935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50005ae686904e35b777d1375074a45e" candidate_uuid: "aa7dfbe8a7334da8944c044ad1b42860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e897257feba3422a9806b63ceabfc4c6"
I20240503 08:09:01.494833 32086 raft_consensus.cc:3037] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:01.494845 31935 raft_consensus.cc:3037] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:01.495537 31935 raft_consensus.cc:2454] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa7dfbe8a7334da8944c044ad1b42860 in term 1.
I20240503 08:09:01.495609 32086 raft_consensus.cc:2454] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa7dfbe8a7334da8944c044ad1b42860 in term 1.
I20240503 08:09:01.495697 31976 leader_election.cc:302] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa7dfbe8a7334da8944c044ad1b42860, e897257feba3422a9806b63ceabfc4c6; no voters:
I20240503 08:09:01.495779 32139 raft_consensus.cc:2781] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:01.495909 32139 raft_consensus.cc:686] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 1 LEADER]: Becoming Leader. State: Replica: aa7dfbe8a7334da8944c044ad1b42860, State: Running, Role: LEADER
I20240503 08:09:01.495957 32139 consensus_queue.cc:237] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } }
I20240503 08:09:01.496480 31832 catalog_manager.cc:5484] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa7dfbe8a7334da8944c044ad1b42860 (127.28.55.66). New cstate: current_term: 1 leader_uuid: "aa7dfbe8a7334da8944c044ad1b42860" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7dfbe8a7334da8944c044ad1b42860" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:02.034152 32146 consensus_queue.cc:1035] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19ae25ffaa8a43ebb69a3339ddc32189" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:02.035949 32149 consensus_queue.cc:1035] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e897257feba3422a9806b63ceabfc4c6" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 33781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:02.180920 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:02.181090 31886 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:02.181206 31886 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:02.181334 31886 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:03.181485 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:04.181527 31886 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:04.181685 31886 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:04.181710 31886 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:04.181847 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:05.182190 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:06.181911 31886 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:06.182086 31886 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:06.182106 31886 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:06.182606 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:07.183007 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:08.182251 31886 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:08.182394 31886 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:08.182428 31886 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:08.183354 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:09.183682 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:10.182627 31886 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:10.182787 31886 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:10.182849 31886 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:10.184024 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:11.184387 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:12.183060 31886 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:12.183216 31886 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:12.183238 31886 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:12.184774 31881 placement_policy_util.cc:407] tablet 50005ae686904e35b777d1375074a45e: detected majority of replicas (2 of 3) at location L0
I20240503 08:09:12.563793 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:09:12.566557 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:12.566721 28893 tablet_replica.cc:330] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6: stopping tablet replica
I20240503 08:09:12.566789 28893 raft_consensus.cc:2229] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:12.566851 28893 raft_consensus.cc:2258] T 50005ae686904e35b777d1375074a45e P e897257feba3422a9806b63ceabfc4c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:12.578778 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:09:12.579845 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:09:12.583031 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:12.583173 28893 tablet_replica.cc:330] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860: stopping tablet replica
I20240503 08:09:12.583245 28893 raft_consensus.cc:2229] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:12.583321 28893 raft_consensus.cc:2258] T 50005ae686904e35b777d1375074a45e P aa7dfbe8a7334da8944c044ad1b42860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:12.584918 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:09:12.585764 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:09:12.588773 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:12.588965 28893 tablet_replica.cc:330] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189: stopping tablet replica
I20240503 08:09:12.589033 28893 raft_consensus.cc:2229] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:12.589084 28893 raft_consensus.cc:2258] T 50005ae686904e35b777d1375074a45e P 19ae25ffaa8a43ebb69a3339ddc32189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:12.600827 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:09:12.601799 28893 master.cc:537] Master@127.28.55.126:41999 shutting down...
I20240503 08:09:12.604142 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:12.604226 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:12.604254 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 25e4f4497aa74b3b83fe172f4434fc69: stopping tablet replica
I20240503 08:09:12.616772 28893 master.cc:552] Master@127.28.55.126:41999 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12458 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240503 08:09:12.621260 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:46853
I20240503 08:09:12.621428 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:12.622622 32162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:12.622623 32160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:12.622885 32159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:12.622915 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:12.623028 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:12.623065 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:12.623077 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723752623077 us; error 0 us; skew 500 ppm
I20240503 08:09:12.628391 28893 webserver.cc:460] Webserver started at http://127.28.55.126:42505/ using document root <none> and password file <none>
I20240503 08:09:12.628484 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:12.628525 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:12.628562 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:12.628829 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "2b393cef270e43ffbd065f91d7add5ed"
format_stamp: "Formatted at 2024-05-03 08:09:12 on dist-test-slave-mlgg"
I20240503 08:09:12.629683 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:12.630142 32167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.630297 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.630350 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "2b393cef270e43ffbd065f91d7add5ed"
format_stamp: "Formatted at 2024-05-03 08:09:12 on dist-test-slave-mlgg"
I20240503 08:09:12.630396 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:12.646463 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:12.646667 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:12.649303 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:46853
I20240503 08:09:12.650995 32219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:46853 every 8 connection(s)
I20240503 08:09:12.651126 32220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:12.652211 32220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed: Bootstrap starting.
I20240503 08:09:12.652551 32220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:12.653676 32220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed: No bootstrap required, opened a new log
I20240503 08:09:12.653806 32220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b393cef270e43ffbd065f91d7add5ed" member_type: VOTER }
I20240503 08:09:12.653872 32220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:12.653885 32220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b393cef270e43ffbd065f91d7add5ed, State: Initialized, Role: FOLLOWER
I20240503 08:09:12.653939 32220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b393cef270e43ffbd065f91d7add5ed" member_type: VOTER }
I20240503 08:09:12.653977 32220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:09:12.653993 32220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:09:12.654016 32220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:12.654563 32220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b393cef270e43ffbd065f91d7add5ed" member_type: VOTER }
I20240503 08:09:12.654626 32220 leader_election.cc:302] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b393cef270e43ffbd065f91d7add5ed; no voters:
I20240503 08:09:12.654737 32220 leader_election.cc:288] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:09:12.654832 32223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:12.654932 32220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:09:12.654968 32223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 1 LEADER]: Becoming Leader. State: Replica: 2b393cef270e43ffbd065f91d7add5ed, State: Running, Role: LEADER
I20240503 08:09:12.655040 32223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b393cef270e43ffbd065f91d7add5ed" member_type: VOTER }
I20240503 08:09:12.655247 32224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b393cef270e43ffbd065f91d7add5ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b393cef270e43ffbd065f91d7add5ed" member_type: VOTER } }
I20240503 08:09:12.655323 32224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:12.655262 32225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b393cef270e43ffbd065f91d7add5ed. Latest consensus state: current_term: 1 leader_uuid: "2b393cef270e43ffbd065f91d7add5ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b393cef270e43ffbd065f91d7add5ed" member_type: VOTER } }
I20240503 08:09:12.655366 32225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:12.655644 32228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:09:12.655805 32228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:09:12.656219 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:09:12.656399 32228 catalog_manager.cc:1347] Generated new cluster ID: 600b73e525d942c989c2fe400287e102
I20240503 08:09:12.656437 32228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:09:12.663616 32228 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:09:12.664099 32228 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:09:12.669509 32228 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed: Generated new TSK 0
I20240503 08:09:12.669603 32228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:09:12.672024 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:12.673424 32241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:12.673491 32244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:12.673475 32242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:12.673615 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:12.673781 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:12.673817 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:12.673835 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723752673835 us; error 0 us; skew 500 ppm
I20240503 08:09:12.674494 28893 webserver.cc:460] Webserver started at http://127.28.55.65:35619/ using document root <none> and password file <none>
I20240503 08:09:12.674580 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:12.674626 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:12.674676 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:12.675010 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "94175e832a7f4428be9adc9cfa9032d9"
format_stamp: "Formatted at 2024-05-03 08:09:12 on dist-test-slave-mlgg"
I20240503 08:09:12.675918 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:12.676366 32249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.676503 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.676543 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "94175e832a7f4428be9adc9cfa9032d9"
format_stamp: "Formatted at 2024-05-03 08:09:12 on dist-test-slave-mlgg"
I20240503 08:09:12.676592 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:12.687166 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:12.687395 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:12.687558 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:12.687759 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:12.687786 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.687819 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:12.687841 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.690593 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:46181
I20240503 08:09:12.690646 32312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:46181 every 8 connection(s)
I20240503 08:09:12.691007 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:12.692309 32317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:12.692404 32320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:12.692314 32318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:12.693169 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:12.693270 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:12.693302 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:12.693318 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723752693319 us; error 0 us; skew 500 ppm
I20240503 08:09:12.694204 28893 webserver.cc:460] Webserver started at http://127.28.55.66:39935/ using document root <none> and password file <none>
I20240503 08:09:12.694299 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:12.694353 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:12.694404 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:12.694754 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "441c21339c5c4941943e09a19ca62fe2"
format_stamp: "Formatted at 2024-05-03 08:09:12 on dist-test-slave-mlgg"
I20240503 08:09:12.694790 32313 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46853
I20240503 08:09:12.694917 32313 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:12.695044 32313 heartbeater.cc:511] Master 127.28.55.126:46853 requested a full tablet report, sending...
I20240503 08:09:12.695287 32184 ts_manager.cc:194] Registered new tserver with Master: 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181)
I20240503 08:09:12.695819 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:12.695899 32184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54600
I20240503 08:09:12.696317 32325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.696424 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.696466 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "441c21339c5c4941943e09a19ca62fe2"
format_stamp: "Formatted at 2024-05-03 08:09:12 on dist-test-slave-mlgg"
I20240503 08:09:12.696516 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:12.709731 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:12.709975 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:12.710120 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:12.710335 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:12.710363 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.710384 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:12.710408 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.713198 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:45189
I20240503 08:09:12.713326 32388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:45189 every 8 connection(s)
I20240503 08:09:12.713574 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:12.714960 32395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:12.714962 32393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:12.715176 32392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:12.715323 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:12.715435 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:12.715468 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:12.715484 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723752715485 us; error 0 us; skew 500 ppm
I20240503 08:09:12.716245 28893 webserver.cc:460] Webserver started at http://127.28.55.67:37151/ using document root <none> and password file <none>
I20240503 08:09:12.716332 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:12.716378 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:12.716477 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:12.716820 32389 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46853
I20240503 08:09:12.716902 32389 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:12.716835 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "e4df5ad2486247e89c2e44b31357188e"
format_stamp: "Formatted at 2024-05-03 08:09:12 on dist-test-slave-mlgg"
I20240503 08:09:12.717037 32389 heartbeater.cc:511] Master 127.28.55.126:46853 requested a full tablet report, sending...
I20240503 08:09:12.717250 32184 ts_manager.cc:194] Registered new tserver with Master: 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:12.717588 32184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54614
I20240503 08:09:12.717830 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:12.718287 32400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.718406 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.718447 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "e4df5ad2486247e89c2e44b31357188e"
format_stamp: "Formatted at 2024-05-03 08:09:12 on dist-test-slave-mlgg"
I20240503 08:09:12.718495 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:12.725219 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:12.725471 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:12.725634 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:12.725832 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:12.725860 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.725888 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:12.725910 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:12.729143 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:44495
I20240503 08:09:12.729177 32463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:44495 every 8 connection(s)
I20240503 08:09:12.732448 32464 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46853
I20240503 08:09:12.732518 32464 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:12.732640 32464 heartbeater.cc:511] Master 127.28.55.126:46853 requested a full tablet report, sending...
I20240503 08:09:12.732898 32184 ts_manager.cc:194] Registered new tserver with Master: e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:12.733255 32184 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54628
I20240503 08:09:12.733745 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004417136s
I20240503 08:09:13.696470 32313 heartbeater.cc:503] Master 127.28.55.126:46853 was elected leader, sending a full tablet report...
I20240503 08:09:13.718104 32389 heartbeater.cc:503] Master 127.28.55.126:46853 was elected leader, sending a full tablet report...
I20240503 08:09:13.733767 32464 heartbeater.cc:503] Master 127.28.55.126:46853 was elected leader, sending a full tablet report...
I20240503 08:09:13.759096 28893 test_util.cc:268] Using random seed: 1010504051
I20240503 08:09:13.764643 32184 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54634:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:09:13.764842 32184 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:09:13.770323 32353 tablet_service.cc:1467] Processing CreateTablet for tablet 22779123b80e4ea4a396192e72b73fe8 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 08:09:13.770399 32350 tablet_service.cc:1467] Processing CreateTablet for tablet 88518b63f713435f9d72e9cf2a395508 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 08:09:13.770548 32350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88518b63f713435f9d72e9cf2a395508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.770673 32353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22779123b80e4ea4a396192e72b73fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.771057 32277 tablet_service.cc:1467] Processing CreateTablet for tablet 22779123b80e4ea4a396192e72b73fe8 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 08:09:13.771163 32277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22779123b80e4ea4a396192e72b73fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.771214 32275 tablet_service.cc:1467] Processing CreateTablet for tablet df0b30e22bc04f638c852c02354109b6 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 08:09:13.771313 32275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0b30e22bc04f638c852c02354109b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.772230 32350 tablet_service.cc:1467] Processing CreateTablet for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 08:09:13.772307 32350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.772537 32353 tablet_service.cc:1467] Processing CreateTablet for tablet b00c72a8ae02442ca1966c3536976263 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 08:09:13.772598 32274 tablet_service.cc:1467] Processing CreateTablet for tablet 88518b63f713435f9d72e9cf2a395508 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 08:09:13.772630 32353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00c72a8ae02442ca1966c3536976263. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.772704 32274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88518b63f713435f9d72e9cf2a395508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.773322 32349 tablet_service.cc:1467] Processing CreateTablet for tablet 4979660e62114184b95e7c2b04a26584 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 08:09:13.773401 32349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4979660e62114184b95e7c2b04a26584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.773849 32273 tablet_service.cc:1467] Processing CreateTablet for tablet 4979660e62114184b95e7c2b04a26584 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 08:09:13.773931 32273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4979660e62114184b95e7c2b04a26584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.774438 32348 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac923b6f4774b589c220467ccb7fd8f (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 08:09:13.774515 32348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac923b6f4774b589c220467ccb7fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.774953 32344 tablet_service.cc:1467] Processing CreateTablet for tablet 2edb5018f69946fdb6ae4a792362ae97 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 08:09:13.775031 32272 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac923b6f4774b589c220467ccb7fd8f (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 08:09:13.775076 32344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edb5018f69946fdb6ae4a792362ae97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.775111 32272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac923b6f4774b589c220467ccb7fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.775242 32347 tablet_service.cc:1467] Processing CreateTablet for tablet 10e0365e2b664690ae51f079c4063b8e (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 08:09:13.775326 32347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e0365e2b664690ae51f079c4063b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.776194 32271 tablet_service.cc:1467] Processing CreateTablet for tablet 10e0365e2b664690ae51f079c4063b8e (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 08:09:13.776257 32271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e0365e2b664690ae51f079c4063b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.776453 32346 tablet_service.cc:1467] Processing CreateTablet for tablet 4eeef2123c7341509085d8ca8ff2b2ad (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 08:09:13.776520 32346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eeef2123c7341509085d8ca8ff2b2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.777115 32270 tablet_service.cc:1467] Processing CreateTablet for tablet 4eeef2123c7341509085d8ca8ff2b2ad (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 08:09:13.777182 32270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eeef2123c7341509085d8ca8ff2b2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.777385 32345 tablet_service.cc:1467] Processing CreateTablet for tablet 7062165e0e3c47c8b43add7f365ec7f4 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 08:09:13.777449 32345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7062165e0e3c47c8b43add7f365ec7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.771164 32276 tablet_service.cc:1467] Processing CreateTablet for tablet 55c3a314149b4921b6cddce536a010e9 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 08:09:13.778031 32276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c3a314149b4921b6cddce536a010e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.770370 32351 tablet_service.cc:1467] Processing CreateTablet for tablet df0b30e22bc04f638c852c02354109b6 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 08:09:13.778272 32351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0b30e22bc04f638c852c02354109b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.778898 32276 tablet_service.cc:1467] Processing CreateTablet for tablet 7062165e0e3c47c8b43add7f365ec7f4 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 08:09:13.770323 32352 tablet_service.cc:1467] Processing CreateTablet for tablet 55c3a314149b4921b6cddce536a010e9 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 08:09:13.778968 32270 tablet_service.cc:1467] Processing CreateTablet for tablet 2edb5018f69946fdb6ae4a792362ae97 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 08:09:13.779101 32352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c3a314149b4921b6cddce536a010e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.778983 32276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7062165e0e3c47c8b43add7f365ec7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.779953 32272 tablet_service.cc:1467] Processing CreateTablet for tablet b00c72a8ae02442ca1966c3536976263 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 08:09:13.780037 32272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00c72a8ae02442ca1966c3536976263. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.780263 32428 tablet_service.cc:1467] Processing CreateTablet for tablet 22779123b80e4ea4a396192e72b73fe8 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 08:09:13.780370 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22779123b80e4ea4a396192e72b73fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.780403 32271 tablet_service.cc:1467] Processing CreateTablet for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 08:09:13.780441 32426 tablet_service.cc:1467] Processing CreateTablet for tablet df0b30e22bc04f638c852c02354109b6 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 08:09:13.780488 32271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.780510 32426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0b30e22bc04f638c852c02354109b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.781169 32484 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.781551 32484 tablet_bootstrap.cc:654] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.781584 32485 tablet_bootstrap.cc:492] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.781616 32428 tablet_service.cc:1467] Processing CreateTablet for tablet b00c72a8ae02442ca1966c3536976263 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 08:09:13.781643 32425 tablet_service.cc:1467] Processing CreateTablet for tablet 88518b63f713435f9d72e9cf2a395508 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 08:09:13.781705 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00c72a8ae02442ca1966c3536976263. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.781774 32424 tablet_service.cc:1467] Processing CreateTablet for tablet 4979660e62114184b95e7c2b04a26584 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 08:09:13.781850 32424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4979660e62114184b95e7c2b04a26584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.782004 32485 tablet_bootstrap.cc:654] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.782140 32419 tablet_service.cc:1467] Processing CreateTablet for tablet 2edb5018f69946fdb6ae4a792362ae97 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 08:09:13.782222 32419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edb5018f69946fdb6ae4a792362ae97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.782377 32270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edb5018f69946fdb6ae4a792362ae97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.783020 32425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88518b63f713435f9d72e9cf2a395508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.781592 32426 tablet_service.cc:1467] Processing CreateTablet for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 08:09:13.783185 32426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.783455 32420 tablet_service.cc:1467] Processing CreateTablet for tablet 7062165e0e3c47c8b43add7f365ec7f4 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 08:09:13.783500 32422 tablet_service.cc:1467] Processing CreateTablet for tablet 10e0365e2b664690ae51f079c4063b8e (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 08:09:13.783550 32420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7062165e0e3c47c8b43add7f365ec7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.783628 32423 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac923b6f4774b589c220467ccb7fd8f (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 08:09:13.783705 32423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac923b6f4774b589c220467ccb7fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.784446 32486 tablet_bootstrap.cc:492] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.780404 32427 tablet_service.cc:1467] Processing CreateTablet for tablet 55c3a314149b4921b6cddce536a010e9 (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 08:09:13.784740 32422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e0365e2b664690ae51f079c4063b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.783452 32421 tablet_service.cc:1467] Processing CreateTablet for tablet 4eeef2123c7341509085d8ca8ff2b2ad (DEFAULT_TABLE table=test-workload [id=79e6b61eb755449b82517db4a1ad4b89]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 08:09:13.784874 32421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eeef2123c7341509085d8ca8ff2b2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.784971 32486 tablet_bootstrap.cc:654] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.784989 32485 tablet_bootstrap.cc:492] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.785118 32485 ts_tablet_manager.cc:1367] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:09:13.785291 32485 raft_consensus.cc:348] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:13.785354 32485 raft_consensus.cc:374] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.785373 32485 raft_consensus.cc:729] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.785403 32485 consensus_queue.cc:260] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:13.785549 32485 ts_tablet_manager.cc:1398] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.785584 32486 tablet_bootstrap.cc:492] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.785619 32486 ts_tablet_manager.cc:1367] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.785782 32486 raft_consensus.cc:348] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:13.785836 32486 raft_consensus.cc:374] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.785852 32486 raft_consensus.cc:729] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.785862 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c3a314149b4921b6cddce536a010e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:13.785876 32486 consensus_queue.cc:260] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:13.785619 32485 tablet_bootstrap.cc:492] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.786162 32486 ts_tablet_manager.cc:1398] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.786223 32486 tablet_bootstrap.cc:492] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.786303 32484 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.786350 32484 ts_tablet_manager.cc:1367] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:09:13.786475 32485 tablet_bootstrap.cc:654] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.786497 32484 raft_consensus.cc:348] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.786550 32486 tablet_bootstrap.cc:654] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.786550 32484 raft_consensus.cc:374] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.786924 32484 raft_consensus.cc:729] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.786999 32484 consensus_queue.cc:260] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.787146 32484 ts_tablet_manager.cc:1398] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.787243 32484 tablet_bootstrap.cc:492] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.787431 32486 tablet_bootstrap.cc:492] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.787487 32486 ts_tablet_manager.cc:1367] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.787611 32485 tablet_bootstrap.cc:492] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.787672 32485 ts_tablet_manager.cc:1367] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:13.787621 32486 raft_consensus.cc:348] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.787767 32486 raft_consensus.cc:374] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.787791 32486 raft_consensus.cc:729] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.787828 32486 consensus_queue.cc:260] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.787882 32485 raft_consensus.cc:348] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.787915 32486 ts_tablet_manager.cc:1398] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.787936 32485 raft_consensus.cc:374] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.787955 32485 raft_consensus.cc:729] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.787969 32486 tablet_bootstrap.cc:492] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.787983 32484 tablet_bootstrap.cc:654] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.787997 32485 consensus_queue.cc:260] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.788075 32485 ts_tablet_manager.cc:1398] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.788133 32485 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.788345 32486 tablet_bootstrap.cc:654] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.788455 32485 tablet_bootstrap.cc:654] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.788995 32485 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.789040 32485 ts_tablet_manager.cc:1367] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.789165 32485 raft_consensus.cc:348] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.789219 32485 raft_consensus.cc:374] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.789237 32485 raft_consensus.cc:729] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.789269 32485 consensus_queue.cc:260] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.789337 32486 tablet_bootstrap.cc:492] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.789374 32485 ts_tablet_manager.cc:1398] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.789378 32486 ts_tablet_manager.cc:1367] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.789431 32484 tablet_bootstrap.cc:492] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.789474 32484 ts_tablet_manager.cc:1367] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:13.789549 32486 raft_consensus.cc:348] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.789439 32485 tablet_bootstrap.cc:492] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.789619 32484 raft_consensus.cc:348] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:13.789721 32484 raft_consensus.cc:374] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.789741 32484 raft_consensus.cc:729] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.789784 32484 consensus_queue.cc:260] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:13.789865 32484 ts_tablet_manager.cc:1398] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.789606 32486 raft_consensus.cc:374] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.789906 32486 raft_consensus.cc:729] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.789949 32486 consensus_queue.cc:260] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.790032 32486 ts_tablet_manager.cc:1398] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:13.790040 32484 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.790089 32486 tablet_bootstrap.cc:492] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.790041 32485 tablet_bootstrap.cc:654] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.790418 32486 tablet_bootstrap.cc:654] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.790418 32484 tablet_bootstrap.cc:654] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.790791 32485 tablet_bootstrap.cc:492] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.790876 32485 ts_tablet_manager.cc:1367] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.791019 32485 raft_consensus.cc:348] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.791072 32485 raft_consensus.cc:374] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.791090 32485 raft_consensus.cc:729] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.791132 32485 consensus_queue.cc:260] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.791210 32485 ts_tablet_manager.cc:1398] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.791260 32485 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.791261 32486 tablet_bootstrap.cc:492] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.791333 32486 ts_tablet_manager.cc:1367] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.791407 32484 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.791445 32484 ts_tablet_manager.cc:1367] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.791455 32486 raft_consensus.cc:348] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.791488 32486 raft_consensus.cc:374] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.791501 32486 raft_consensus.cc:729] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.791534 32486 consensus_queue.cc:260] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.791566 32484 raft_consensus.cc:348] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.791630 32484 raft_consensus.cc:374] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.791621 32485 tablet_bootstrap.cc:654] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.791630 32486 ts_tablet_manager.cc:1398] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.791651 32484 raft_consensus.cc:729] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.791715 32486 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.791743 32484 consensus_queue.cc:260] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.791836 32484 ts_tablet_manager.cc:1398] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.791888 32484 tablet_bootstrap.cc:492] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.792069 32486 tablet_bootstrap.cc:654] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.792220 32484 tablet_bootstrap.cc:654] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.792435 32485 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.792487 32485 ts_tablet_manager.cc:1367] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.792624 32485 raft_consensus.cc:348] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.792681 32485 raft_consensus.cc:374] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.792701 32485 raft_consensus.cc:729] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.792727 32485 consensus_queue.cc:260] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.792804 32485 ts_tablet_manager.cc:1398] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.792934 32485 tablet_bootstrap.cc:492] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.793260 32485 tablet_bootstrap.cc:654] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.793417 32486 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.793469 32486 ts_tablet_manager.cc:1367] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:13.793658 32486 raft_consensus.cc:348] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.793777 32486 raft_consensus.cc:374] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.793812 32486 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.793809 32484 tablet_bootstrap.cc:492] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.793907 32484 ts_tablet_manager.cc:1367] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:13.793954 32485 tablet_bootstrap.cc:492] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.793905 32486 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.794000 32485 ts_tablet_manager.cc:1367] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.794095 32486 ts_tablet_manager.cc:1398] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.794121 32485 raft_consensus.cc:348] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.794080 32484 raft_consensus.cc:348] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.794178 32485 raft_consensus.cc:374] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.794201 32485 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.794215 32486 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.794240 32485 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.794314 32485 ts_tablet_manager.cc:1398] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.794359 32485 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.794698 32485 tablet_bootstrap.cc:654] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.794714 32486 tablet_bootstrap.cc:654] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.794183 32484 raft_consensus.cc:374] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.794889 32484 raft_consensus.cc:729] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.794932 32484 consensus_queue.cc:260] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.795017 32484 ts_tablet_manager.cc:1398] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:13.795073 32484 tablet_bootstrap.cc:492] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.795296 32486 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.795331 32486 ts_tablet_manager.cc:1367] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.795367 32484 tablet_bootstrap.cc:654] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.795444 32486 raft_consensus.cc:348] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.795485 32486 raft_consensus.cc:374] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.795497 32486 raft_consensus.cc:729] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.795531 32486 consensus_queue.cc:260] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.795593 32486 ts_tablet_manager.cc:1398] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.795639 32486 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.795986 32486 tablet_bootstrap.cc:654] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.796262 32484 tablet_bootstrap.cc:492] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.796303 32484 ts_tablet_manager.cc:1367] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.796424 32485 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.796468 32485 ts_tablet_manager.cc:1367] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:09:13.796577 32486 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.796622 32486 ts_tablet_manager.cc:1367] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.796633 32485 raft_consensus.cc:348] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.796695 32485 raft_consensus.cc:374] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.796720 32485 raft_consensus.cc:729] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.796752 32486 raft_consensus.cc:348] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.796766 32485 consensus_queue.cc:260] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.796427 32484 raft_consensus.cc:348] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.796861 32485 ts_tablet_manager.cc:1398] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.796880 32484 raft_consensus.cc:374] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.796897 32484 raft_consensus.cc:729] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.796914 32485 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.796937 32484 consensus_queue.cc:260] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.797016 32484 ts_tablet_manager.cc:1398] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:13.797075 32484 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.797292 32485 tablet_bootstrap.cc:654] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.797437 32484 tablet_bootstrap.cc:654] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.797602 32486 raft_consensus.cc:374] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.797670 32486 raft_consensus.cc:729] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.797756 32486 consensus_queue.cc:260] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.797863 32485 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.797904 32485 ts_tablet_manager.cc:1367] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.797993 32486 ts_tablet_manager.cc:1398] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.798051 32486 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.798041 32485 raft_consensus.cc:348] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.798116 32485 raft_consensus.cc:374] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.798135 32485 raft_consensus.cc:729] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.798171 32485 consensus_queue.cc:260] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.798243 32485 ts_tablet_manager.cc:1398] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.798321 32485 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.798404 32486 tablet_bootstrap.cc:654] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.798657 32485 tablet_bootstrap.cc:654] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.799113 32484 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.799168 32484 ts_tablet_manager.cc:1367] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:13.799237 32485 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.799283 32485 ts_tablet_manager.cc:1367] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.799299 32484 raft_consensus.cc:348] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.799357 32484 raft_consensus.cc:374] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.799381 32484 raft_consensus.cc:729] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.799422 32484 consensus_queue.cc:260] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.799412 32485 raft_consensus.cc:348] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.799487 32485 raft_consensus.cc:374] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.799512 32484 ts_tablet_manager.cc:1398] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:09:13.799512 32485 raft_consensus.cc:729] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.799589 32484 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.799599 32485 consensus_queue.cc:260] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.799818 32485 ts_tablet_manager.cc:1398] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:09:13.799922 32485 tablet_bootstrap.cc:492] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.799939 32484 tablet_bootstrap.cc:654] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.800346 32485 tablet_bootstrap.cc:654] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.800366 32486 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.800478 32486 ts_tablet_manager.cc:1367] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:09:13.800629 32486 raft_consensus.cc:348] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.800676 32484 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.800724 32484 ts_tablet_manager.cc:1367] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.800686 32486 raft_consensus.cc:374] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.800837 32486 raft_consensus.cc:729] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.800851 32484 raft_consensus.cc:348] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.800897 32484 raft_consensus.cc:374] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.800879 32486 consensus_queue.cc:260] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.800923 32484 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.800938 32485 tablet_bootstrap.cc:492] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.800959 32484 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.800976 32486 ts_tablet_manager.cc:1398] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.800976 32485 ts_tablet_manager.cc:1367] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.801044 32486 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.801048 32484 ts_tablet_manager.cc:1398] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.801120 32484 tablet_bootstrap.cc:492] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.801136 32485 raft_consensus.cc:348] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.801191 32485 raft_consensus.cc:374] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.801209 32485 raft_consensus.cc:729] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.801263 32485 consensus_queue.cc:260] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.801373 32485 ts_tablet_manager.cc:1398] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.801424 32485 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.801446 32484 tablet_bootstrap.cc:654] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.801429 32486 tablet_bootstrap.cc:654] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.801806 32485 tablet_bootstrap.cc:654] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.802057 32484 tablet_bootstrap.cc:492] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.802106 32484 ts_tablet_manager.cc:1367] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.802258 32484 raft_consensus.cc:348] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.802314 32484 raft_consensus.cc:374] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.802337 32484 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.802384 32484 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.802317 32486 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.802445 32486 ts_tablet_manager.cc:1367] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.802471 32484 ts_tablet_manager.cc:1398] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.802531 32484 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.802573 32486 raft_consensus.cc:348] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.802660 32486 raft_consensus.cc:374] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.802680 32486 raft_consensus.cc:729] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.802723 32486 consensus_queue.cc:260] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.802848 32486 ts_tablet_manager.cc:1398] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.802901 32486 tablet_bootstrap.cc:492] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.802922 32484 tablet_bootstrap.cc:654] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.803251 32486 tablet_bootstrap.cc:654] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.803539 32484 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.803572 32484 ts_tablet_manager.cc:1367] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.802567 32485 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.803653 32485 ts_tablet_manager.cc:1367] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:09:13.803700 32484 raft_consensus.cc:348] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.803759 32484 raft_consensus.cc:374] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.803776 32484 raft_consensus.cc:729] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.803793 32486 tablet_bootstrap.cc:492] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.803857 32486 ts_tablet_manager.cc:1367] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.803844 32485 raft_consensus.cc:348] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.803854 32484 consensus_queue.cc:260] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.803941 32485 raft_consensus.cc:374] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.803964 32485 raft_consensus.cc:729] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.803992 32484 ts_tablet_manager.cc:1398] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.804020 32486 raft_consensus.cc:348] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.804052 32484 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.804085 32486 raft_consensus.cc:374] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.804102 32486 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.804136 32486 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.804215 32486 ts_tablet_manager.cc:1398] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.804260 32486 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.804425 32484 tablet_bootstrap.cc:654] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.804611 32486 tablet_bootstrap.cc:654] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.804993 32484 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.804014 32485 consensus_queue.cc:260] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.805083 32484 ts_tablet_manager.cc:1367] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.805151 32486 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.805193 32486 ts_tablet_manager.cc:1367] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.805256 32484 raft_consensus.cc:348] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.805289 32485 ts_tablet_manager.cc:1398] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:09:13.805310 32484 raft_consensus.cc:374] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.805316 32486 raft_consensus.cc:348] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.805356 32484 raft_consensus.cc:729] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.805362 32486 raft_consensus.cc:374] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.805395 32486 raft_consensus.cc:729] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.805410 32484 consensus_queue.cc:260] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.805471 32486 consensus_queue.cc:260] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.805369 32485 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: Bootstrap starting.
I20240503 08:09:13.805572 32486 ts_tablet_manager.cc:1398] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.805680 32486 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e: Bootstrap starting.
I20240503 08:09:13.805958 32485 tablet_bootstrap.cc:654] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.806051 32486 tablet_bootstrap.cc:654] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.805572 32484 ts_tablet_manager.cc:1398] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.806356 32484 tablet_bootstrap.cc:492] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.806545 32485 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: No bootstrap required, opened a new log
I20240503 08:09:13.806615 32485 ts_tablet_manager.cc:1367] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.806749 32484 tablet_bootstrap.cc:654] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.806778 32485 raft_consensus.cc:348] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.806921 32485 raft_consensus.cc:374] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.806960 32485 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.807025 32486 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e: No bootstrap required, opened a new log
I20240503 08:09:13.807026 32485 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.807101 32486 ts_tablet_manager.cc:1367] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:13.807186 32485 ts_tablet_manager.cc:1398] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:13.807247 32486 raft_consensus.cc:348] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.807313 32486 raft_consensus.cc:374] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.807361 32486 raft_consensus.cc:729] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.807401 32484 tablet_bootstrap.cc:492] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.807421 32486 consensus_queue.cc:260] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.807448 32484 ts_tablet_manager.cc:1367] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.807516 32486 ts_tablet_manager.cc:1398] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.807575 32484 raft_consensus.cc:348] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.807637 32484 raft_consensus.cc:374] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.807725 32484 raft_consensus.cc:729] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.807864 32484 consensus_queue.cc:260] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.808117 32484 ts_tablet_manager.cc:1398] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.808251 32484 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: Bootstrap starting.
I20240503 08:09:13.808624 32484 tablet_bootstrap.cc:654] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:13.809126 32484 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: No bootstrap required, opened a new log
I20240503 08:09:13.809166 32484 ts_tablet_manager.cc:1367] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:13.809291 32484 raft_consensus.cc:348] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.809335 32484 raft_consensus.cc:374] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:13.809351 32484 raft_consensus.cc:729] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Initialized, Role: FOLLOWER
I20240503 08:09:13.809391 32484 consensus_queue.cc:260] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.809468 32484 ts_tablet_manager.cc:1398] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:13.809666 32491 raft_consensus.cc:482] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.809736 32491 raft_consensus.cc:504] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.809901 32491 leader_election.cc:288] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.812644 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.812714 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2" is_pre_election: true
I20240503 08:09:13.812750 32287 raft_consensus.cc:2454] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.812798 32363 raft_consensus.cc:2454] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.812940 32401 leader_election.cc:302] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.813021 32491 raft_consensus.cc:2781] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.813081 32491 raft_consensus.cc:482] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.813100 32491 raft_consensus.cc:3037] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.813700 32491 raft_consensus.cc:504] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.813823 32491 leader_election.cc:288] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.813987 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:13.814013 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.814076 32363 raft_consensus.cc:3037] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.814093 32287 raft_consensus.cc:3037] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.814787 32363 raft_consensus.cc:2454] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.814787 32287 raft_consensus.cc:2454] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.815001 32403 leader_election.cc:302] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.815111 32491 raft_consensus.cc:2781] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.815251 32491 raft_consensus.cc:686] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:13.815299 32491 consensus_queue.cc:237] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.815865 32182 catalog_manager.cc:5484] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:13.834828 32494 raft_consensus.cc:482] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.834955 32494 raft_consensus.cc:504] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.835207 32494 leader_election.cc:288] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.838707 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c3a314149b4921b6cddce536a010e9" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e" is_pre_election: true
I20240503 08:09:13.838864 32438 raft_consensus.cc:2454] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 0.
I20240503 08:09:13.839027 32327 leader_election.cc:302] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.839031 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c3a314149b4921b6cddce536a010e9" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.839135 32494 raft_consensus.cc:2781] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.839166 32287 raft_consensus.cc:2454] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 0.
I20240503 08:09:13.839181 32494 raft_consensus.cc:482] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.839221 32494 raft_consensus.cc:3037] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.839906 32494 raft_consensus.cc:504] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.840040 32494 leader_election.cc:288] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.840236 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c3a314149b4921b6cddce536a010e9" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.840257 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c3a314149b4921b6cddce536a010e9" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:13.840317 32287 raft_consensus.cc:3037] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.840323 32438 raft_consensus.cc:3037] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.840999 32287 raft_consensus.cc:2454] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 1.
I20240503 08:09:13.840999 32438 raft_consensus.cc:2454] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 1.
I20240503 08:09:13.841234 32328 leader_election.cc:302] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:13.841342 32494 raft_consensus.cc:2781] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.841513 32494 raft_consensus.cc:686] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:13.841567 32494 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.842232 32182 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:13.842484 32491 raft_consensus.cc:482] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.842540 32491 raft_consensus.cc:504] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.842649 32491 leader_election.cc:288] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.842793 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.842866 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2" is_pre_election: true
I20240503 08:09:13.842952 32363 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.842878 32287 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.843091 32403 leader_election.cc:302] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.843165 32491 raft_consensus.cc:2781] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.843212 32491 raft_consensus.cc:482] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.843230 32491 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.843734 32491 raft_consensus.cc:504] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.843842 32491 leader_election.cc:288] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.844005 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.844020 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:13.844092 32287 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.844094 32363 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.844697 32363 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.844714 32287 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.844888 32403 leader_election.cc:302] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.845036 32491 raft_consensus.cc:2781] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.845096 32491 raft_consensus.cc:686] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:13.845149 32491 consensus_queue.cc:237] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.845734 32182 catalog_manager.cc:5484] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:13.868613 32528 raft_consensus.cc:482] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.868743 32528 raft_consensus.cc:504] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.868930 32528 leader_election.cc:288] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.869171 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2" is_pre_election: true
I20240503 08:09:13.869174 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.869309 32287 raft_consensus.cc:2454] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.869308 32363 raft_consensus.cc:2454] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.869484 32401 leader_election.cc:302] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.869573 32528 raft_consensus.cc:2781] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.869612 32528 raft_consensus.cc:482] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.869630 32528 raft_consensus.cc:3037] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.870371 32528 raft_consensus.cc:504] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.870503 32528 leader_election.cc:288] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.870673 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.870699 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:13.870752 32287 raft_consensus.cc:3037] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.870760 32363 raft_consensus.cc:3037] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.871348 32363 raft_consensus.cc:2454] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.871352 32287 raft_consensus.cc:2454] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.871518 32403 leader_election.cc:302] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.871604 32528 raft_consensus.cc:2781] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.871641 32528 raft_consensus.cc:686] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:13.871691 32528 consensus_queue.cc:237] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.872324 32182 catalog_manager.cc:5484] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:13.897010 32494 raft_consensus.cc:482] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.897133 32494 raft_consensus.cc:504] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.897333 32494 leader_election.cc:288] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.897563 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac923b6f4774b589c220467ccb7fd8f" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.897572 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac923b6f4774b589c220467ccb7fd8f" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e" is_pre_election: true
I20240503 08:09:13.897665 32287 raft_consensus.cc:2454] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 0.
I20240503 08:09:13.897686 32438 raft_consensus.cc:2454] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 0.
I20240503 08:09:13.897835 32328 leader_election.cc:302] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.897928 32494 raft_consensus.cc:2781] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.897965 32494 raft_consensus.cc:482] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.897984 32494 raft_consensus.cc:3037] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.898767 32494 raft_consensus.cc:504] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.898936 32494 leader_election.cc:288] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.899093 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac923b6f4774b589c220467ccb7fd8f" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.899155 32287 raft_consensus.cc:3037] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.899121 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac923b6f4774b589c220467ccb7fd8f" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:13.899204 32438 raft_consensus.cc:3037] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.899703 32287 raft_consensus.cc:2454] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 1.
I20240503 08:09:13.899787 32438 raft_consensus.cc:2454] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 1.
I20240503 08:09:13.899881 32328 leader_election.cc:302] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:13.899981 32494 raft_consensus.cc:2781] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.900024 32494 raft_consensus.cc:686] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:13.900072 32494 consensus_queue.cc:237] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.900676 32182 catalog_manager.cc:5484] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:13.917227 32494 raft_consensus.cc:482] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.917335 32494 raft_consensus.cc:504] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:13.917495 32494 leader_election.cc:288] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.917732 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e" is_pre_election: true
I20240503 08:09:13.917744 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.917878 32438 raft_consensus.cc:2454] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 0.
I20240503 08:09:13.917881 32287 raft_consensus.cc:2454] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 0.
I20240503 08:09:13.918030 32328 leader_election.cc:302] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:13.918135 32494 raft_consensus.cc:2781] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.918177 32494 raft_consensus.cc:482] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.918195 32494 raft_consensus.cc:3037] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.918839 32494 raft_consensus.cc:504] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:13.918985 32494 leader_election.cc:288] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.919144 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.919183 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:13.919234 32287 raft_consensus.cc:3037] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.919245 32438 raft_consensus.cc:3037] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.919879 32287 raft_consensus.cc:2454] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 1.
I20240503 08:09:13.919879 32438 raft_consensus.cc:2454] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 1.
I20240503 08:09:13.920105 32328 leader_election.cc:302] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:13.920202 32494 raft_consensus.cc:2781] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.920267 32494 raft_consensus.cc:686] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:13.920323 32494 consensus_queue.cc:237] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:13.920961 32182 catalog_manager.cc:5484] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:13.926084 32528 raft_consensus.cc:482] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.926180 32528 raft_consensus.cc:504] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.926344 32528 leader_election.cc:288] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.926548 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.926617 32287 raft_consensus.cc:2454] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.926573 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2" is_pre_election: true
I20240503 08:09:13.926678 32363 raft_consensus.cc:2454] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.926765 32403 leader_election.cc:302] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.926944 32528 raft_consensus.cc:2781] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.926986 32528 raft_consensus.cc:482] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.927079 32528 raft_consensus.cc:3037] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.927764 32528 raft_consensus.cc:504] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.927903 32528 leader_election.cc:288] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.928072 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.928125 32287 raft_consensus.cc:3037] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.928100 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:13.928158 32363 raft_consensus.cc:3037] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.928640 32287 raft_consensus.cc:2454] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.928706 32363 raft_consensus.cc:2454] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.928826 32403 leader_election.cc:302] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.928910 32528 raft_consensus.cc:2781] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.928973 32528 raft_consensus.cc:686] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:13.929021 32528 consensus_queue.cc:237] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.929651 32182 catalog_manager.cc:5484] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:13.956199 32528 raft_consensus.cc:482] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.956324 32528 raft_consensus.cc:504] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.956498 32528 leader_election.cc:288] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.956647 32494 raft_consensus.cc:482] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.956741 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.956743 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2" is_pre_election: true
I20240503 08:09:13.956840 32287 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.956966 32363 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.956763 32494 raft_consensus.cc:504] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.957116 32403 leader_election.cc:302] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.957115 32494 leader_election.cc:288] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.957162 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7062165e0e3c47c8b43add7f365ec7f4" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.957242 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7062165e0e3c47c8b43add7f365ec7f4" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e" is_pre_election: true
I20240503 08:09:13.957281 32528 raft_consensus.cc:2781] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.957258 32287 raft_consensus.cc:2454] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 0.
I20240503 08:09:13.957309 32438 raft_consensus.cc:2454] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 0.
I20240503 08:09:13.957310 32528 raft_consensus.cc:482] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.957360 32528 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.957432 32328 leader_election.cc:302] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:13.957518 32494 raft_consensus.cc:2781] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.957551 32494 raft_consensus.cc:482] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.957567 32494 raft_consensus.cc:3037] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.958076 32528 raft_consensus.cc:504] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.958096 32494 raft_consensus.cc:504] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.958250 32528 leader_election.cc:288] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.958273 32494 leader_election.cc:288] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.958371 32286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7062165e0e3c47c8b43add7f365ec7f4" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.958359 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.958417 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:13.958436 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7062165e0e3c47c8b43add7f365ec7f4" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:13.958485 32363 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.958498 32438 raft_consensus.cc:3037] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.958472 32286 raft_consensus.cc:3037] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.959062 32438 raft_consensus.cc:2454] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 1.
I20240503 08:09:13.959236 32327 leader_election.cc:302] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.959208 32286 raft_consensus.cc:2454] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 1.
I20240503 08:09:13.959326 32494 raft_consensus.cc:2781] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.959367 32494 raft_consensus.cc:686] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:13.958477 32287 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.959445 32494 consensus_queue.cc:237] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.959959 32287 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.959208 32363 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.960181 32403 leader_election.cc:302] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.960261 32528 raft_consensus.cc:2781] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.960222 32182 catalog_manager.cc:5484] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:13.960314 32528 raft_consensus.cc:686] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:13.960354 32528 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.960969 32184 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:13.971940 32528 raft_consensus.cc:482] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.972059 32528 raft_consensus.cc:504] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.972237 32528 leader_election.cc:288] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.972460 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:13.972533 32287 raft_consensus.cc:2454] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.972482 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2" is_pre_election: true
I20240503 08:09:13.972574 32363 raft_consensus.cc:2454] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:13.972659 32403 leader_election.cc:302] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.972736 32528 raft_consensus.cc:2781] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.972785 32528 raft_consensus.cc:482] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.972815 32528 raft_consensus.cc:3037] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.973407 32528 raft_consensus.cc:504] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.973532 32528 leader_election.cc:288] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:13.973681 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:13.973712 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:13.973762 32363 raft_consensus.cc:3037] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.973789 32287 raft_consensus.cc:3037] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.974433 32363 raft_consensus.cc:2454] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.974434 32287 raft_consensus.cc:2454] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:13.974630 32403 leader_election.cc:302] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:13.974771 32528 raft_consensus.cc:2781] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.974849 32528 raft_consensus.cc:686] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:13.974903 32528 consensus_queue.cc:237] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.975560 32184 catalog_manager.cc:5484] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:13.989055 32490 raft_consensus.cc:482] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:13.989179 32490 raft_consensus.cc:504] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.989393 32490 leader_election.cc:288] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.992640 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2" is_pre_election: true
I20240503 08:09:13.992658 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e" is_pre_election: true
I20240503 08:09:13.992790 32363 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 0.
I20240503 08:09:13.992798 32438 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 0.
I20240503 08:09:13.992949 32250 leader_election.cc:302] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:13.993042 32490 raft_consensus.cc:2781] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:13.993083 32490 raft_consensus.cc:482] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:13.993103 32490 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.993780 32490 raft_consensus.cc:504] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.993916 32490 leader_election.cc:288] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 1 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:13.994069 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:13.994084 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:13.994138 32363 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.994174 32438 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:13.994668 32438 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 1.
I20240503 08:09:13.994663 32363 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 1.
I20240503 08:09:13.994865 32250 leader_election.cc:302] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:13.994952 32490 raft_consensus.cc:2781] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:13.995114 32490 raft_consensus.cc:686] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:13.995177 32490 consensus_queue.cc:237] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:13.995759 32184 catalog_manager.cc:5484] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 1 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:14.091416 32528 raft_consensus.cc:482] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:14.091532 32528 raft_consensus.cc:504] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.091714 32528 leader_election.cc:288] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:14.091956 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2" is_pre_election: true
I20240503 08:09:14.091981 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9" is_pre_election: true
I20240503 08:09:14.092065 32363 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:14.092074 32287 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 0.
I20240503 08:09:14.092250 32401 leader_election.cc:302] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:14.092377 32528 raft_consensus.cc:2781] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:14.092425 32528 raft_consensus.cc:482] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:14.092445 32528 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:14.093061 32528 raft_consensus.cc:504] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.093191 32528 leader_election.cc:288] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:14.093369 32287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:14.093396 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:14.093457 32287 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:14.093461 32363 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:14.094105 32363 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:14.094105 32287 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 1.
I20240503 08:09:14.094296 32401 leader_election.cc:302] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:14.094388 32528 raft_consensus.cc:2781] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:14.094436 32528 raft_consensus.cc:686] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:14.094472 32528 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.095175 32184 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:14.134604 32363 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.134773 32438 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.134896 32437 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.135025 32532 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.135157 32528 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.135169 32362 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.135257 32438 raft_consensus.cc:1261] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.135365 32437 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.135468 32528 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.135475 32532 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.135668 32532 consensus_queue.cc:1035] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.135743 32532 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.136065 32287 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.136170 32286 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.136677 32361 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.136786 32360 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.136972 32360 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.137112 32491 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.137202 32491 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.137383 32361 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.138171 32361 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.138392 32528 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.138499 32528 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.139098 32357 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.139194 32491 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.139247 32434 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.139416 32490 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.139897 32286 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.139998 32287 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.140128 32491 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.140178 32285 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.140226 32491 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.140242 32283 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.140280 32284 raft_consensus.cc:1261] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.140345 32282 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.140465 32281 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.140689 32287 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.140774 32286 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:14.141469 32544 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.141652 32491 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.141690 32528 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.141747 32491 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.141752 32528 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.141793 32588 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:09:14.143234 32559 mvcc.cc:205] Tried to move back new op lower bound from 7023508496938291200 to 7023508495735652352. Current Snapshot: MvccSnapshot[applied={T|T < 7023508496934428672}]
I20240503 08:09:14.145707 32494 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.145839 32494 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.145906 32494 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.145956 32494 consensus_queue.cc:1035] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:14.147615 32554 mvcc.cc:205] Tried to move back new op lower bound from 7023508496949084160 to 7023508495627952128. Current Snapshot: MvccSnapshot[applied={T|T < 7023508496941326336}]
I20240503 08:09:14.150738 32557 mvcc.cc:205] Tried to move back new op lower bound from 7023508496949211136 to 7023508496093818880. Current Snapshot: MvccSnapshot[applied={T|T < 7023508496944668672}]
I20240503 08:09:14.233217 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:14.234557 32640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:14.234619 32641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:14.234619 32643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:14.234879 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:14.234987 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:14.235018 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:14.235035 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723754235035 us; error 0 us; skew 500 ppm
I20240503 08:09:14.235865 28893 webserver.cc:460] Webserver started at http://127.28.55.68:45143/ using document root <none> and password file <none>
I20240503 08:09:14.235958 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:14.236018 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:14.236068 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:14.236347 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-3-root/instance:
uuid: "15ce703959504369884c5eac0c7fa77d"
format_stamp: "Formatted at 2024-05-03 08:09:14 on dist-test-slave-mlgg"
I20240503 08:09:14.237200 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:14.237637 32648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.237777 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.237831 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-3-root
uuid: "15ce703959504369884c5eac0c7fa77d"
format_stamp: "Formatted at 2024-05-03 08:09:14 on dist-test-slave-mlgg"
I20240503 08:09:14.237886 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:14.248828 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:14.249089 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:14.249279 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:14.249481 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:14.249509 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.249531 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:14.249554 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.253120 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.68:43117
I20240503 08:09:14.253259 32711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.68:43117 every 8 connection(s)
I20240503 08:09:14.253574 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:14.254921 32716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:14.254941 32718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:14.254964 32715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:14.255400 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:14.255492 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:14.255522 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:14.255538 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723754255538 us; error 0 us; skew 500 ppm
I20240503 08:09:14.256587 28893 webserver.cc:460] Webserver started at http://127.28.55.69:38937/ using document root <none> and password file <none>
I20240503 08:09:14.256680 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:14.256757 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:14.256842 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:14.257151 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-4-root/instance:
uuid: "08561681ad9a4dc5be7c769e0a959ca9"
format_stamp: "Formatted at 2024-05-03 08:09:14 on dist-test-slave-mlgg"
I20240503 08:09:14.257361 32712 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46853
I20240503 08:09:14.257426 32712 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:14.257560 32712 heartbeater.cc:511] Master 127.28.55.126:46853 requested a full tablet report, sending...
I20240503 08:09:14.257823 32183 ts_manager.cc:194] Registered new tserver with Master: 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117)
I20240503 08:09:14.258200 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:14.258265 32183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54642
I20240503 08:09:14.258718 32723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.258857 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.258906 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-4-root
uuid: "08561681ad9a4dc5be7c769e0a959ca9"
format_stamp: "Formatted at 2024-05-03 08:09:14 on dist-test-slave-mlgg"
I20240503 08:09:14.258957 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:14.263157 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:14.263594 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:14.263727 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:14.263921 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:14.263949 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.263969 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:14.263993 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.267369 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.69:45733
I20240503 08:09:14.267556 318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.69:45733 every 8 connection(s)
I20240503 08:09:14.267781 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:14.269150 323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:14.269243 322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:14.269191 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:14.269834 325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:14.269989 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:14.270025 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:14.270042 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723754270043 us; error 0 us; skew 500 ppm
I20240503 08:09:14.271034 28893 webserver.cc:460] Webserver started at http://127.28.55.70:38581/ using document root <none> and password file <none>
I20240503 08:09:14.271205 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:14.271288 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:14.271340 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:14.271620 319 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46853
I20240503 08:09:14.271677 319 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:14.271682 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-5-root/instance:
uuid: "2ac031e1b6da412da3f1c277da38d38b"
format_stamp: "Formatted at 2024-05-03 08:09:14 on dist-test-slave-mlgg"
I20240503 08:09:14.271832 319 heartbeater.cc:511] Master 127.28.55.126:46853 requested a full tablet report, sending...
I20240503 08:09:14.272061 32183 ts_manager.cc:194] Registered new tserver with Master: 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733)
I20240503 08:09:14.272430 32183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54652
I20240503 08:09:14.272691 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:14.273145 330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.273249 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.273295 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-5-root
uuid: "2ac031e1b6da412da3f1c277da38d38b"
format_stamp: "Formatted at 2024-05-03 08:09:14 on dist-test-slave-mlgg"
I20240503 08:09:14.273344 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:14.279951 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:14.280182 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:14.280328 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:14.280539 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:14.280568 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.280598 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:14.280622 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:14.284076 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.70:33543
I20240503 08:09:14.284303 393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.70:33543 every 8 connection(s)
I20240503 08:09:14.287608 394 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46853
I20240503 08:09:14.287678 394 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:14.287775 394 heartbeater.cc:511] Master 127.28.55.126:46853 requested a full tablet report, sending...
I20240503 08:09:14.288008 32183 ts_manager.cc:194] Registered new tserver with Master: 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543)
I20240503 08:09:14.288331 32183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54668
I20240503 08:09:14.656332 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:14.661358 32436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:14.661463 32436 raft_consensus.cc:595] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:14.661486 32438 consensus_queue.cc:237] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:14.661804 32436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4979660e62114184b95e7c2b04a26584"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:14.661871 32436 raft_consensus.cc:595] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:14.662145 32362 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 08:09:14.662212 32436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:14.662292 32436 raft_consensus.cc:595] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:14.662322 32281 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 08:09:14.662438 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:09:14.662500 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:14.662506 32637 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:14.662649 32636 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:14.663301 32634 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:14.663429 32281 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:14.664748 32183 catalog_manager.cc:5484] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index -1 to 25, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:14.665325 32362 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
W20240503 08:09:14.665585 32402 consensus_peers.cc:485] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e -> Peer 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733): Couldn't send request to peer 08561681ad9a4dc5be7c769e0a959ca9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07d3a63eb2f640bc9f1ea96a6e071bd5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:14.666673 32436 consensus_queue.cc:237] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:14.667171 32281 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 08:09:14.667259 32362 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 08:09:14.667344 32636 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:14.667457 32636 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:14.667886 32635 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:14.667970 32281 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:14.669087 32184 catalog_manager.cc:5484] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index -1 to 25, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:14.669236 32362 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
W20240503 08:09:14.670751 32401 consensus_peers.cc:485] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e -> Peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117): Couldn't send request to peer 15ce703959504369884c5eac0c7fa77d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eeef2123c7341509085d8ca8ff2b2ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:14.672683 32362 consensus_queue.cc:237] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:14.673354 32436 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:14.673579 32624 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:14.673595 32281 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:14.673816 32624 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:14.674223 32639 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:14.674518 32283 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:14.675642 32183 catalog_manager.cc:5484] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index -1 to 26, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:14.676020 32436 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:14.676391 32362 consensus_queue.cc:237] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
W20240503 08:09:14.676893 32327 consensus_peers.cc:485] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 -> Peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543): Couldn't send request to peer 2ac031e1b6da412da3f1c277da38d38b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ac923b6f4774b589c220467ccb7fd8f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:14.676939 32283 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 08:09:14.677165 32608 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:14.677172 32436 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 08:09:14.677407 32608 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:14.677757 32624 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:14.677989 32436 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:14.678949 32181 catalog_manager.cc:5484] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index -1 to 25, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:14.678779 32283 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:14.680423 32362 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:14.680934 32436 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:14.681115 32281 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:14.681102 32624 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:14.681494 32624 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:14.682221 32623 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
W20240503 08:09:14.682365 32327 consensus_peers.cc:485] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 -> Peer 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733): Couldn't send request to peer 08561681ad9a4dc5be7c769e0a959ca9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7062165e0e3c47c8b43add7f365ec7f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:14.682433 32281 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:14.683506 32436 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:14.683501 32184 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index -1 to 26, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:14.683766 32438 consensus_queue.cc:237] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:14.684360 32281 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:14.684360 32362 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:14.684628 32636 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:14.685012 32636 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:14.685192 32632 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
W20240503 08:09:14.685376 32326 consensus_peers.cc:485] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 -> Peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117): Couldn't send request to peer 15ce703959504369884c5eac0c7fa77d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55c3a314149b4921b6cddce536a010e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:14.685750 32362 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:14.686460 32281 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:14.687294 32181 catalog_manager.cc:5484] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index -1 to 26, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
W20240503 08:09:14.689155 32402 consensus_peers.cc:485] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e -> Peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543): Couldn't send request to peer 2ac031e1b6da412da3f1c277da38d38b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88518b63f713435f9d72e9cf2a395508. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:14.742559 32636 raft_consensus.cc:979] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:14.742655 32636 raft_consensus.cc:1067] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:14.742835 32281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4979660e62114184b95e7c2b04a26584"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49104
I20240503 08:09:14.742949 32281 raft_consensus.cc:482] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:14.742980 32281 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:14.743698 32281 raft_consensus.cc:504] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.743868 32281 leader_election.cc:288] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:14.743968 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:14.743983 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:14.744060 32362 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:14.744072 32436 raft_consensus.cc:3032] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:14.744153 32436 raft_consensus.cc:729] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:14.744237 32436 consensus_queue.cc:260] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.744357 32436 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:14.744876 32362 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:14.745023 32250 leader_election.cc:302] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:14.745110 32613 raft_consensus.cc:2781] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:14.745114 32436 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:14.745230 32613 raft_consensus.cc:686] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:14.745282 32613 consensus_queue.cc:237] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.745738 32184 catalog_manager.cc:5484] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 1 to 2, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:14.765333 32635 raft_consensus.cc:979] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:14.765446 32635 raft_consensus.cc:1067] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:14.765604 32281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49104
I20240503 08:09:14.765712 32281 raft_consensus.cc:482] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:14.765745 32281 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:14.766498 32281 raft_consensus.cc:504] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.766662 32281 leader_election.cc:288] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:14.766794 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:14.766914 32436 raft_consensus.cc:3032] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:14.766837 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:14.766957 32436 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:14.766991 32362 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:14.767060 32436 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.767182 32436 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:14.767690 32362 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:14.767796 32436 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:14.767874 32250 leader_election.cc:302] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:14.767969 32613 raft_consensus.cc:2781] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:14.768014 32613 raft_consensus.cc:686] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:14.768064 32613 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.768543 32184 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 1 to 2, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:14.836680 32636 raft_consensus.cc:979] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:14.836788 32636 raft_consensus.cc:1067] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:14.836994 32281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49104
I20240503 08:09:14.837101 32281 raft_consensus.cc:482] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:14.837144 32281 raft_consensus.cc:3037] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:14.838030 32281 raft_consensus.cc:504] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.838188 32281 leader_election.cc:288] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:14.838342 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:14.838431 32436 raft_consensus.cc:3032] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:14.838462 32436 raft_consensus.cc:729] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:14.838517 32436 consensus_queue.cc:260] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.838611 32436 raft_consensus.cc:3037] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:14.839376 32436 raft_consensus.cc:2454] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:14.838342 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:14.839543 32251 leader_election.cc:302] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:14.839680 32613 raft_consensus.cc:2781] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:14.839586 32362 raft_consensus.cc:3037] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:14.839741 32613 raft_consensus.cc:686] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:14.839799 32613 consensus_queue.cc:237] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:14.840314 32184 catalog_manager.cc:5484] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 1 to 2, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:14.840412 32362 raft_consensus.cc:2454] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:15.065611 413 ts_tablet_manager.cc:902] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: Initiating tablet copy from peer 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:15.065953 413 tablet_copy_client.cc:308] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:45189
I20240503 08:09:15.068915 32373 tablet_copy_service.cc:140] P 441c21339c5c4941943e09a19ca62fe2: Received BeginTabletCopySession request for tablet 7062165e0e3c47c8b43add7f365ec7f4 from peer 08561681ad9a4dc5be7c769e0a959ca9 ({username='slave'} at 127.0.0.1:50280)
I20240503 08:09:15.068986 32373 tablet_copy_service.cc:161] P 441c21339c5c4941943e09a19ca62fe2: Beginning new tablet copy session on tablet 7062165e0e3c47c8b43add7f365ec7f4 from peer 08561681ad9a4dc5be7c769e0a959ca9 at {username='slave'} at 127.0.0.1:50280: session id = 08561681ad9a4dc5be7c769e0a959ca9-7062165e0e3c47c8b43add7f365ec7f4
I20240503 08:09:15.069721 32373 tablet_copy_source_session.cc:205] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:15.070204 413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7062165e0e3c47c8b43add7f365ec7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:15.071949 413 tablet_copy_client.cc:783] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:15.072094 413 tablet_copy_client.cc:647] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:15.072746 417 tablet_copy_client.cc:1002] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:15.196552 419 ts_tablet_manager.cc:902] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: Initiating tablet copy from peer 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:15.196979 419 tablet_copy_client.cc:308] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:45189
I20240503 08:09:15.199069 425 ts_tablet_manager.cc:902] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: Initiating tablet copy from peer e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:15.199421 425 tablet_copy_client.cc:308] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.67:44495
I20240503 08:09:15.200807 32373 tablet_copy_service.cc:140] P 441c21339c5c4941943e09a19ca62fe2: Received BeginTabletCopySession request for tablet 7ac923b6f4774b589c220467ccb7fd8f from peer 2ac031e1b6da412da3f1c277da38d38b ({username='slave'} at 127.0.0.1:50282)
I20240503 08:09:15.200881 32373 tablet_copy_service.cc:161] P 441c21339c5c4941943e09a19ca62fe2: Beginning new tablet copy session on tablet 7ac923b6f4774b589c220467ccb7fd8f from peer 2ac031e1b6da412da3f1c277da38d38b at {username='slave'} at 127.0.0.1:50282: session id = 2ac031e1b6da412da3f1c277da38d38b-7ac923b6f4774b589c220467ccb7fd8f
I20240503 08:09:15.201457 32373 tablet_copy_source_session.cc:205] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:15.201817 419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac923b6f4774b589c220467ccb7fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:15.203414 419 tablet_copy_client.cc:783] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:15.203541 419 tablet_copy_client.cc:647] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:15.203752 32448 tablet_copy_service.cc:140] P e4df5ad2486247e89c2e44b31357188e: Received BeginTabletCopySession request for tablet 88518b63f713435f9d72e9cf2a395508 from peer 2ac031e1b6da412da3f1c277da38d38b ({username='slave'} at 127.0.0.1:57392)
I20240503 08:09:15.203802 32448 tablet_copy_service.cc:161] P e4df5ad2486247e89c2e44b31357188e: Beginning new tablet copy session on tablet 88518b63f713435f9d72e9cf2a395508 from peer 2ac031e1b6da412da3f1c277da38d38b at {username='slave'} at 127.0.0.1:57392: session id = 2ac031e1b6da412da3f1c277da38d38b-88518b63f713435f9d72e9cf2a395508
I20240503 08:09:15.204094 423 tablet_copy_client.cc:1002] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:15.204250 32448 tablet_copy_source_session.cc:205] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:15.204541 425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88518b63f713435f9d72e9cf2a395508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:15.205474 425 tablet_copy_client.cc:783] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:15.205587 425 tablet_copy_client.cc:647] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:15.206156 429 tablet_copy_client.cc:1002] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:15.218451 32362 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240503 08:09:15.218734 412 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 08:09:15.220155 32438 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20240503 08:09:15.220443 32613 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 08:09:15.237537 435 ts_tablet_manager.cc:902] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: Initiating tablet copy from peer 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:15.237952 435 tablet_copy_client.cc:308] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:45189
I20240503 08:09:15.240795 32373 tablet_copy_service.cc:140] P 441c21339c5c4941943e09a19ca62fe2: Received BeginTabletCopySession request for tablet 55c3a314149b4921b6cddce536a010e9 from peer 15ce703959504369884c5eac0c7fa77d ({username='slave'} at 127.0.0.1:50294)
I20240503 08:09:15.240870 32373 tablet_copy_service.cc:161] P 441c21339c5c4941943e09a19ca62fe2: Beginning new tablet copy session on tablet 55c3a314149b4921b6cddce536a010e9 from peer 15ce703959504369884c5eac0c7fa77d at {username='slave'} at 127.0.0.1:50294: session id = 15ce703959504369884c5eac0c7fa77d-55c3a314149b4921b6cddce536a010e9
I20240503 08:09:15.241374 32373 tablet_copy_source_session.cc:205] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:15.241700 435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c3a314149b4921b6cddce536a010e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:15.243400 435 tablet_copy_client.cc:783] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:15.243536 435 tablet_copy_client.cc:647] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:15.244097 439 tablet_copy_client.cc:1002] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:15.245086 441 ts_tablet_manager.cc:902] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: Initiating tablet copy from peer e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:15.245417 441 tablet_copy_client.cc:308] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.67:44495
I20240503 08:09:15.248225 32448 tablet_copy_service.cc:140] P e4df5ad2486247e89c2e44b31357188e: Received BeginTabletCopySession request for tablet 4eeef2123c7341509085d8ca8ff2b2ad from peer 15ce703959504369884c5eac0c7fa77d ({username='slave'} at 127.0.0.1:57406)
I20240503 08:09:15.248293 32448 tablet_copy_service.cc:161] P e4df5ad2486247e89c2e44b31357188e: Beginning new tablet copy session on tablet 4eeef2123c7341509085d8ca8ff2b2ad from peer 15ce703959504369884c5eac0c7fa77d at {username='slave'} at 127.0.0.1:57406: session id = 15ce703959504369884c5eac0c7fa77d-4eeef2123c7341509085d8ca8ff2b2ad
I20240503 08:09:15.248911 32448 tablet_copy_source_session.cc:205] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:15.249229 441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eeef2123c7341509085d8ca8ff2b2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:15.250869 441 tablet_copy_client.cc:783] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:15.251009 441 tablet_copy_client.cc:647] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:15.251761 443 tablet_copy_client.cc:1002] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:15.258857 32712 heartbeater.cc:503] Master 127.28.55.126:46853 was elected leader, sending a full tablet report...
I20240503 08:09:15.272972 319 heartbeater.cc:503] Master 127.28.55.126:46853 was elected leader, sending a full tablet report...
I20240503 08:09:15.284098 446 ts_tablet_manager.cc:902] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: Initiating tablet copy from peer e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:15.284452 446 tablet_copy_client.cc:308] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.67:44495
I20240503 08:09:15.287562 32448 tablet_copy_service.cc:140] P e4df5ad2486247e89c2e44b31357188e: Received BeginTabletCopySession request for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 from peer 08561681ad9a4dc5be7c769e0a959ca9 ({username='slave'} at 127.0.0.1:57418)
I20240503 08:09:15.287657 32448 tablet_copy_service.cc:161] P e4df5ad2486247e89c2e44b31357188e: Beginning new tablet copy session on tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 from peer 08561681ad9a4dc5be7c769e0a959ca9 at {username='slave'} at 127.0.0.1:57418: session id = 08561681ad9a4dc5be7c769e0a959ca9-07d3a63eb2f640bc9f1ea96a6e071bd5
I20240503 08:09:15.288137 32448 tablet_copy_source_session.cc:205] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:15.288479 446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:15.288756 394 heartbeater.cc:503] Master 127.28.55.126:46853 was elected leader, sending a full tablet report...
I20240503 08:09:15.289961 446 tablet_copy_client.cc:783] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:15.290086 446 tablet_copy_client.cc:647] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:15.290608 450 tablet_copy_client.cc:1002] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:15.325637 32436 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:15.325930 32613 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:15.327224 32362 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:15.327425 452 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:15.346890 32436 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:09:15.347149 452 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:15.347669 32362 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:09:15.347838 32613 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:16.662699 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:16.663456 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:16.663637 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:17.072996 413 tablet_copy_client.cc:523] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:17.074398 413 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap starting.
I20240503 08:09:17.078764 413 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:17.079030 413 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap complete.
I20240503 08:09:17.079165 413 ts_tablet_manager.cc:1367] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.003s
I20240503 08:09:17.079334 413 raft_consensus.cc:348] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:17.079416 413 raft_consensus.cc:729] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08561681ad9a4dc5be7c769e0a959ca9, State: Initialized, Role: LEARNER
I20240503 08:09:17.079445 413 consensus_queue.cc:260] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:17.079594 413 ts_tablet_manager.cc:1398] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:09:17.079814 32373 tablet_copy_service.cc:342] P 441c21339c5c4941943e09a19ca62fe2: Request end of tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-7062165e0e3c47c8b43add7f365ec7f4 received from {username='slave'} at 127.0.0.1:50280
I20240503 08:09:17.079883 32373 tablet_copy_service.cc:433] P 441c21339c5c4941943e09a19ca62fe2: ending tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-7062165e0e3c47c8b43add7f365ec7f4 on tablet 7062165e0e3c47c8b43add7f365ec7f4 with peer 08561681ad9a4dc5be7c769e0a959ca9
I20240503 08:09:17.204319 419 tablet_copy_client.cc:523] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:17.205461 419 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap starting.
I20240503 08:09:17.206362 425 tablet_copy_client.cc:523] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:17.207147 425 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap starting.
I20240503 08:09:17.209141 419 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:17.209321 419 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap complete.
I20240503 08:09:17.209442 419 ts_tablet_manager.cc:1367] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:09:17.209623 419 raft_consensus.cc:348] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:17.209698 419 raft_consensus.cc:729] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ac031e1b6da412da3f1c277da38d38b, State: Initialized, Role: LEARNER
I20240503 08:09:17.209728 419 consensus_queue.cc:260] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:17.209869 419 ts_tablet_manager.cc:1398] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: Time spent starting tablet: real 0.000s user 0.000s sys 0.003s
I20240503 08:09:17.210134 32373 tablet_copy_service.cc:342] P 441c21339c5c4941943e09a19ca62fe2: Request end of tablet copy session 2ac031e1b6da412da3f1c277da38d38b-7ac923b6f4774b589c220467ccb7fd8f received from {username='slave'} at 127.0.0.1:50282
I20240503 08:09:17.210202 32373 tablet_copy_service.cc:433] P 441c21339c5c4941943e09a19ca62fe2: ending tablet copy session 2ac031e1b6da412da3f1c277da38d38b-7ac923b6f4774b589c220467ccb7fd8f on tablet 7ac923b6f4774b589c220467ccb7fd8f with peer 2ac031e1b6da412da3f1c277da38d38b
I20240503 08:09:17.211076 425 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:17.211223 425 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap complete.
I20240503 08:09:17.211314 425 ts_tablet_manager.cc:1367] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:09:17.211438 425 raft_consensus.cc:348] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:17.211495 425 raft_consensus.cc:729] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ac031e1b6da412da3f1c277da38d38b, State: Initialized, Role: LEARNER
I20240503 08:09:17.211521 425 consensus_queue.cc:260] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:17.211596 425 ts_tablet_manager.cc:1398] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:17.211807 32448 tablet_copy_service.cc:342] P e4df5ad2486247e89c2e44b31357188e: Request end of tablet copy session 2ac031e1b6da412da3f1c277da38d38b-88518b63f713435f9d72e9cf2a395508 received from {username='slave'} at 127.0.0.1:57392
I20240503 08:09:17.211863 32448 tablet_copy_service.cc:433] P e4df5ad2486247e89c2e44b31357188e: ending tablet copy session 2ac031e1b6da412da3f1c277da38d38b-88518b63f713435f9d72e9cf2a395508 on tablet 88518b63f713435f9d72e9cf2a395508 with peer 2ac031e1b6da412da3f1c277da38d38b
I20240503 08:09:17.244370 435 tablet_copy_client.cc:523] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:17.245577 435 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: Bootstrap starting.
I20240503 08:09:17.249614 435 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=89 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:17.249864 435 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: Bootstrap complete.
I20240503 08:09:17.249995 435 ts_tablet_manager.cc:1367] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:09:17.250233 435 raft_consensus.cc:348] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:17.250329 435 raft_consensus.cc:729] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 15ce703959504369884c5eac0c7fa77d, State: Initialized, Role: LEARNER
I20240503 08:09:17.250363 435 consensus_queue.cc:260] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:17.250514 435 ts_tablet_manager.cc:1398] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:17.250730 32373 tablet_copy_service.cc:342] P 441c21339c5c4941943e09a19ca62fe2: Request end of tablet copy session 15ce703959504369884c5eac0c7fa77d-55c3a314149b4921b6cddce536a010e9 received from {username='slave'} at 127.0.0.1:50294
I20240503 08:09:17.250787 32373 tablet_copy_service.cc:433] P 441c21339c5c4941943e09a19ca62fe2: ending tablet copy session 15ce703959504369884c5eac0c7fa77d-55c3a314149b4921b6cddce536a010e9 on tablet 55c3a314149b4921b6cddce536a010e9 with peer 15ce703959504369884c5eac0c7fa77d
I20240503 08:09:17.251968 441 tablet_copy_client.cc:523] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:17.252686 441 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: Bootstrap starting.
I20240503 08:09:17.255877 441 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=92 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:17.256086 441 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: Bootstrap complete.
I20240503 08:09:17.256198 441 ts_tablet_manager.cc:1367] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:09:17.256357 441 raft_consensus.cc:348] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:17.256426 441 raft_consensus.cc:729] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 15ce703959504369884c5eac0c7fa77d, State: Initialized, Role: LEARNER
I20240503 08:09:17.256455 441 consensus_queue.cc:260] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:17.256570 441 ts_tablet_manager.cc:1398] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20240503 08:09:17.256793 32448 tablet_copy_service.cc:342] P e4df5ad2486247e89c2e44b31357188e: Request end of tablet copy session 15ce703959504369884c5eac0c7fa77d-4eeef2123c7341509085d8ca8ff2b2ad received from {username='slave'} at 127.0.0.1:57406
I20240503 08:09:17.256842 32448 tablet_copy_service.cc:433] P e4df5ad2486247e89c2e44b31357188e: ending tablet copy session 15ce703959504369884c5eac0c7fa77d-4eeef2123c7341509085d8ca8ff2b2ad on tablet 4eeef2123c7341509085d8ca8ff2b2ad with peer 15ce703959504369884c5eac0c7fa77d
I20240503 08:09:17.267474 368 raft_consensus.cc:1203] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 08:09:17.290879 446 tablet_copy_client.cc:523] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:17.291879 446 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap starting.
I20240503 08:09:17.295603 446 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:17.295838 446 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap complete.
I20240503 08:09:17.295957 446 ts_tablet_manager.cc:1367] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:09:17.296124 446 raft_consensus.cc:348] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:17.296216 446 raft_consensus.cc:729] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08561681ad9a4dc5be7c769e0a959ca9, State: Initialized, Role: LEARNER
I20240503 08:09:17.296249 446 consensus_queue.cc:260] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:17.296502 446 ts_tablet_manager.cc:1398] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s
I20240503 08:09:17.296768 32448 tablet_copy_service.cc:342] P e4df5ad2486247e89c2e44b31357188e: Request end of tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-07d3a63eb2f640bc9f1ea96a6e071bd5 received from {username='slave'} at 127.0.0.1:57418
I20240503 08:09:17.296835 32448 tablet_copy_service.cc:433] P e4df5ad2486247e89c2e44b31357188e: ending tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-07d3a63eb2f640bc9f1ea96a6e071bd5 on tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 with peer 08561681ad9a4dc5be7c769e0a959ca9
I20240503 08:09:17.463196 32761 raft_consensus.cc:1203] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240503 08:09:17.586962 32686 raft_consensus.cc:1203] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 08:09:17.637914 368 raft_consensus.cc:1203] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 08:09:17.754544 471 raft_consensus.cc:1050] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2: attempting to promote NON_VOTER 2ac031e1b6da412da3f1c277da38d38b to VOTER
I20240503 08:09:17.754925 471 consensus_queue.cc:237] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:17.755484 368 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:17.755491 32438 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:17.755532 32281 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:17.755729 457 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:17.755854 470 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:17.756014 473 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:17.756804 470 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:17.756917 32438 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:17.757205 368 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:17.758215 32281 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:17.758350 32184 catalog_manager.cc:5484] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b reported cstate change: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:17.759970 32360 consensus_queue.cc:237] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:17.760448 32281 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:09:17.760499 368 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:09:17.760649 473 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:17.760897 471 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:17.761186 473 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:17.761284 32281 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:17.762323 32181 catalog_manager.cc:5484] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index 27 to 28, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:17.762501 32168 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7ac923b6f4774b589c220467ccb7fd8f with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:17.762964 368 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:17.764359 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ac923b6f4774b589c220467ccb7fd8f with delete_type TABLET_DATA_TOMBSTONED (TS e4df5ad2486247e89c2e44b31357188e not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:57340
I20240503 08:09:17.764622 481 tablet_replica.cc:330] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:17.764863 481 raft_consensus.cc:2229] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:17.764981 481 raft_consensus.cc:2258] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:17.765285 481 ts_tablet_manager.cc:1875] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:17.766927 481 ts_tablet_manager.cc:1888] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 08:09:17.766992 481 log.cc:1196] T 7ac923b6f4774b589c220467ccb7fd8f P e4df5ad2486247e89c2e44b31357188e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/7ac923b6f4774b589c220467ccb7fd8f
I20240503 08:09:17.767310 32169 catalog_manager.cc:4830] TS e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495): tablet 7ac923b6f4774b589c220467ccb7fd8f (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:17.853017 32686 raft_consensus.cc:1203] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240503 08:09:17.855684 32761 raft_consensus.cc:1203] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240503 08:09:17.904168 473 raft_consensus.cc:1050] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: attempting to promote NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 to VOTER
I20240503 08:09:17.904426 473 consensus_queue.cc:237] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:17.905193 32281 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:17.905248 32438 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:17.905381 32761 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 LEARNER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:17.905452 471 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:17.905620 457 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:17.905727 470 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:17.906718 457 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:17.906863 32436 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:17.908164 32181 catalog_manager.cc:5484] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:17.908217 32281 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:17.908488 32761 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:17.909936 32362 consensus_queue.cc:237] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:17.910323 32761 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:17.910346 32281 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:17.910578 471 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:17.910718 473 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:17.911281 470 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:17.911396 32281 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:17.911445 32761 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:17.912360 32168 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7062165e0e3c47c8b43add7f365ec7f4 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:17.912422 32181 catalog_manager.cc:5484] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index 26 to 27, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:17.913245 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7062165e0e3c47c8b43add7f365ec7f4 with delete_type TABLET_DATA_TOMBSTONED (TS e4df5ad2486247e89c2e44b31357188e not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:57340
I20240503 08:09:17.913349 481 tablet_replica.cc:330] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:17.913412 481 raft_consensus.cc:2229] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:17.913494 481 raft_consensus.cc:2258] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:17.913872 481 ts_tablet_manager.cc:1875] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:17.915683 481 ts_tablet_manager.cc:1888] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240503 08:09:17.915745 481 log.cc:1196] T 7062165e0e3c47c8b43add7f365ec7f4 P e4df5ad2486247e89c2e44b31357188e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/7062165e0e3c47c8b43add7f365ec7f4
I20240503 08:09:17.916189 32169 catalog_manager.cc:4830] TS e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495): tablet 7062165e0e3c47c8b43add7f365ec7f4 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:18.142328 459 raft_consensus.cc:1050] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e: attempting to promote NON_VOTER 2ac031e1b6da412da3f1c277da38d38b to VOTER
I20240503 08:09:18.142560 459 consensus_queue.cc:237] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:18.143296 32362 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.143360 368 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.143296 32281 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.143632 470 raft_consensus.cc:1050] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: attempting to promote NON_VOTER 15ce703959504369884c5eac0c7fa77d to VOTER
I20240503 08:09:18.143620 489 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.143775 489 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.143813 470 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:18.143909 486 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.144448 32283 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.144455 32686 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 1 LEARNER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.144666 471 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.144726 489 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:18.144942 32436 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.144945 471 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.145226 471 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.145680 32362 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:18.146021 470 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.146135 32686 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.146891 32182 catalog_manager.cc:5484] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:18.147127 368 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:18.147594 32183 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 26 to 27, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.148068 32281 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.148147 32283 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:18.148351 32436 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.149978 32436 consensus_queue.cc:237] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:18.150084 32362 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:18.150431 368 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:09:18.150413 32686 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:09:18.150431 32283 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:09:18.150628 471 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:18.150647 488 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:18.151008 32436 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:09:18.151026 488 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:18.151199 471 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:18.151297 489 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:18.151449 470 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.151562 32686 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.152395 32168 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55c3a314149b4921b6cddce536a010e9 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:18.152516 32184 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 27 to 28, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.152618 32180 catalog_manager.cc:5484] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index 27 to 28, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:18.152725 32169 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88518b63f713435f9d72e9cf2a395508 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:18.152896 32283 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:18.153398 32352 tablet_service.cc:1514] Processing DeleteTablet for tablet 88518b63f713435f9d72e9cf2a395508 with delete_type TABLET_DATA_TOMBSTONED (TS 441c21339c5c4941943e09a19ca62fe2 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:18.153523 32270 tablet_service.cc:1514] Processing DeleteTablet for tablet 55c3a314149b4921b6cddce536a010e9 with delete_type TABLET_DATA_TOMBSTONED (TS 94175e832a7f4428be9adc9cfa9032d9 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:49090
I20240503 08:09:18.153648 498 tablet_replica.cc:330] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:18.153757 498 raft_consensus.cc:2229] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:18.153788 499 tablet_replica.cc:330] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:18.153978 499 raft_consensus.cc:2229] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:18.154053 499 raft_consensus.cc:2258] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:18.153890 498 raft_consensus.cc:2258] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:18.154384 499 ts_tablet_manager.cc:1875] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:18.154670 498 ts_tablet_manager.cc:1875] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:18.154505 32434 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.154986 368 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:18.156282 499 ts_tablet_manager.cc:1888] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 08:09:18.156416 499 log.cc:1196] T 55c3a314149b4921b6cddce536a010e9 P 94175e832a7f4428be9adc9cfa9032d9: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/55c3a314149b4921b6cddce536a010e9
I20240503 08:09:18.156764 32170 catalog_manager.cc:4830] TS 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181): tablet 55c3a314149b4921b6cddce536a010e9 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:18.157035 498 ts_tablet_manager.cc:1888] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 08:09:18.157092 498 log.cc:1196] T 88518b63f713435f9d72e9cf2a395508 P 441c21339c5c4941943e09a19ca62fe2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/88518b63f713435f9d72e9cf2a395508
I20240503 08:09:18.157413 32168 catalog_manager.cc:4830] TS 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189): tablet 88518b63f713435f9d72e9cf2a395508 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:18.250593 486 raft_consensus.cc:1050] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: attempting to promote NON_VOTER 15ce703959504369884c5eac0c7fa77d to VOTER
I20240503 08:09:18.250993 486 consensus_queue.cc:237] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:18.251760 32283 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:18.251819 32362 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:18.251907 32686 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 1 LEARNER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:18.252389 459 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:18.252612 488 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:18.252769 489 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:18.253537 489 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.253685 32686 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.253782 32283 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.254926 32181 catalog_manager.cc:5484] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:18.255401 32362 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.256702 32434 consensus_queue.cc:237] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:18.257123 32283 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.257218 32686 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.257346 459 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.257449 459 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.257995 459 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.258121 32686 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.258991 32283 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.259142 32184 catalog_manager.cc:5484] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 26 to 27, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:18.259980 32352 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eeef2123c7341509085d8ca8ff2b2ad with delete_type TABLET_DATA_TOMBSTONED (TS 441c21339c5c4941943e09a19ca62fe2 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:18.260022 32169 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4eeef2123c7341509085d8ca8ff2b2ad with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:18.260094 498 tablet_replica.cc:330] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:18.260167 498 raft_consensus.cc:2229] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:18.260253 498 raft_consensus.cc:2258] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:18.260519 498 ts_tablet_manager.cc:1875] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:18.261933 498 ts_tablet_manager.cc:1888] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240503 08:09:18.261987 498 log.cc:1196] T 4eeef2123c7341509085d8ca8ff2b2ad P 441c21339c5c4941943e09a19ca62fe2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/4eeef2123c7341509085d8ca8ff2b2ad
I20240503 08:09:18.262315 32168 catalog_manager.cc:4830] TS 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189): tablet 4eeef2123c7341509085d8ca8ff2b2ad (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:18.361290 486 raft_consensus.cc:1050] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: attempting to promote NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 to VOTER
I20240503 08:09:18.361541 486 consensus_queue.cc:237] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:18.362077 32761 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 LEARNER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:18.362077 32283 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:18.362123 32362 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 08:09:18.362354 459 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:18.362495 459 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:18.362608 488 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:18.363375 489 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:18.363483 32283 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:18.364631 32184 catalog_manager.cc:5484] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:18.364697 32761 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:18.365346 32362 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:18.366436 32436 consensus_queue.cc:237] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:18.366854 32761 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.366873 32362 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 08:09:18.367033 459 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.367130 489 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:18.367673 486 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:18.367815 32761 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:18.368675 32169 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:18.368798 32184 catalog_manager.cc:5484] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 reported cstate change: config changed from index 26 to 27, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New cstate: current_term: 1 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:18.369020 32362 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:18.369688 32270 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 with delete_type TABLET_DATA_TOMBSTONED (TS 94175e832a7f4428be9adc9cfa9032d9 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:49090
I20240503 08:09:18.369796 499 tablet_replica.cc:330] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:18.369864 499 raft_consensus.cc:2229] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:18.369933 499 raft_consensus.cc:2258] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:18.370184 499 ts_tablet_manager.cc:1875] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:18.371260 499 ts_tablet_manager.cc:1888] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240503 08:09:18.371309 499 log.cc:1196] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 94175e832a7f4428be9adc9cfa9032d9: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/07d3a63eb2f640bc9f1ea96a6e071bd5
I20240503 08:09:18.371596 32170 catalog_manager.cc:4830] TS 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181): tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:18.663863 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:18.664371 32434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:18.664464 32434 raft_consensus.cc:595] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:18.667647 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:18.667737 32283 raft_consensus.cc:595] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:18.670352 32362 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55c3a314149b4921b6cddce536a010e9"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "15ce703959504369884c5eac0c7fa77d"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:18.670432 32362 raft_consensus.cc:595] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Received request to transfer leadership to TS 15ce703959504369884c5eac0c7fa77d
I20240503 08:09:18.670748 32362 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ac923b6f4774b589c220467ccb7fd8f"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "2ac031e1b6da412da3f1c277da38d38b"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:18.670796 32362 raft_consensus.cc:595] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Received request to transfer leadership to TS 2ac031e1b6da412da3f1c277da38d38b
I20240503 08:09:18.671142 32436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:18.671214 32436 raft_consensus.cc:595] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:18.671509 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:18.671557 32283 raft_consensus.cc:595] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:18.671669 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240503 08:09:18.671732 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:18.765089 32613 raft_consensus.cc:979] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:18.765182 32613 raft_consensus.cc:1067] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:18.765390 32362 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50240
I20240503 08:09:18.765463 32362 raft_consensus.cc:482] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:18.765482 32362 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:18.766117 32362 raft_consensus.cc:504] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:18.766242 32362 leader_election.cc:288] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 2 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:18.766456 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:18.766494 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:18.766562 32283 raft_consensus.cc:3032] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:18.766578 32436 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:18.766593 32283 raft_consensus.cc:729] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:18.766676 32283 consensus_queue.cc:260] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:18.766760 32283 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:18.767382 32436 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 2.
I20240503 08:09:18.767382 32283 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 2.
I20240503 08:09:18.767598 32327 leader_election.cc:302] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:18.767686 470 raft_consensus.cc:2781] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:18.767781 470 raft_consensus.cc:686] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:18.767820 470 consensus_queue.cc:237] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:18.768415 32181 catalog_manager.cc:5484] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 1 to 2, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 2 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:18.781877 470 raft_consensus.cc:979] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower 2ac031e1b6da412da3f1c277da38d38b to start an election
I20240503 08:09:18.781970 470 raft_consensus.cc:1067] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Signalling peer 2ac031e1b6da412da3f1c277da38d38b to start an election
I20240503 08:09:18.782181 368 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ac923b6f4774b589c220467ccb7fd8f"
dest_uuid: "2ac031e1b6da412da3f1c277da38d38b"
from {username='slave'} at 127.0.0.1:59336
I20240503 08:09:18.782269 368 raft_consensus.cc:482] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:18.782289 368 raft_consensus.cc:3037] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:18.783000 368 raft_consensus.cc:504] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:18.783157 368 leader_election.cc:288] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [CANDIDATE]: Term 2 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:18.783354 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac923b6f4774b589c220467ccb7fd8f" candidate_uuid: "2ac031e1b6da412da3f1c277da38d38b" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:18.783428 32362 raft_consensus.cc:3032] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:18.783452 32362 raft_consensus.cc:729] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:18.783499 32362 consensus_queue.cc:260] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:18.783614 32362 raft_consensus.cc:3037] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:18.784307 32362 raft_consensus.cc:2454] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac031e1b6da412da3f1c277da38d38b in term 2.
I20240503 08:09:18.784546 331 leader_election.cc:302] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac031e1b6da412da3f1c277da38d38b, 441c21339c5c4941943e09a19ca62fe2; no voters:
I20240503 08:09:18.784787 515 raft_consensus.cc:2781] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:18.784911 515 raft_consensus.cc:686] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEADER]: Becoming Leader. State: Replica: 2ac031e1b6da412da3f1c277da38d38b, State: Running, Role: LEADER
I20240503 08:09:18.784979 515 consensus_queue.cc:237] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:18.785540 32181 catalog_manager.cc:5484] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b reported cstate change: term changed from 1 to 2, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70). New cstate: current_term: 2 leader_uuid: "2ac031e1b6da412da3f1c277da38d38b" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:18.786232 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac923b6f4774b589c220467ccb7fd8f" candidate_uuid: "2ac031e1b6da412da3f1c277da38d38b" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:18.786322 32283 raft_consensus.cc:3037] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:18.786944 32283 raft_consensus.cc:2454] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac031e1b6da412da3f1c277da38d38b in term 2.
I20240503 08:09:18.894924 32613 raft_consensus.cc:979] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:18.895018 32613 raft_consensus.cc:1067] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:18.895263 32362 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50240
I20240503 08:09:18.895360 32362 raft_consensus.cc:482] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:18.895383 32362 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:18.896100 32362 raft_consensus.cc:504] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:18.896250 32362 leader_election.cc:288] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 3 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:18.896409 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:18.896407 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:18.896503 32283 raft_consensus.cc:3032] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:09:18.896515 32436 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:18.896520 32283 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:18.896629 32283 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:18.896714 32283 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:18.897167 32436 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 3.
I20240503 08:09:18.897315 32283 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 3.
I20240503 08:09:18.897416 32327 leader_election.cc:302] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:18.897550 470 raft_consensus.cc:2781] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:09:18.897609 470 raft_consensus.cc:686] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:18.897660 470 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:18.898206 32181 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 2 to 3, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 3 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:18.987394 489 raft_consensus.cc:979] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:18.987489 489 raft_consensus.cc:1067] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:18.987679 32283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49104
I20240503 08:09:18.987768 32283 raft_consensus.cc:482] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:18.987797 32283 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:18.988463 32283 raft_consensus.cc:504] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:18.988615 32283 leader_election.cc:288] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:18.988785 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:18.988842 32436 raft_consensus.cc:3032] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:18.988827 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:18.988889 32436 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:18.988927 32362 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:18.988970 32436 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:18.989048 32436 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:18.989565 32436 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:18.989567 32362 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:18.989791 32251 leader_election.cc:302] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:18.989884 32613 raft_consensus.cc:2781] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:18.989948 32613 raft_consensus.cc:686] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:18.989982 32613 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:18.990514 32181 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 1 to 2, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:19.083595 489 raft_consensus.cc:979] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:19.083693 489 raft_consensus.cc:1067] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:19.083920 32283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49104
I20240503 08:09:19.084002 32283 raft_consensus.cc:482] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:19.084031 32283 raft_consensus.cc:3037] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:19.084689 32283 raft_consensus.cc:504] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:19.084839 32283 leader_election.cc:288] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Requested vote from peers e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495), 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543)
I20240503 08:09:19.084987 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:19.085047 32436 raft_consensus.cc:3032] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:19.085068 32436 raft_consensus.cc:729] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:19.085114 32436 consensus_queue.cc:260] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:19.085215 32436 raft_consensus.cc:3037] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:19.085958 32436 raft_consensus.cc:2454] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:19.086138 32251 leader_election.cc:302] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:19.086232 32613 raft_consensus.cc:2781] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:19.086303 32613 raft_consensus.cc:686] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:19.086354 32613 consensus_queue.cc:237] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:19.086863 32181 catalog_manager.cc:5484] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 1 to 2, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:19.087980 368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "2ac031e1b6da412da3f1c277da38d38b"
I20240503 08:09:19.088047 368 raft_consensus.cc:3037] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:19.088606 368 raft_consensus.cc:2454] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:19.169529 470 raft_consensus.cc:979] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower 15ce703959504369884c5eac0c7fa77d to start an election
I20240503 08:09:19.169638 470 raft_consensus.cc:1067] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Signalling peer 15ce703959504369884c5eac0c7fa77d to start an election
I20240503 08:09:19.169817 32686 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55c3a314149b4921b6cddce536a010e9"
dest_uuid: "15ce703959504369884c5eac0c7fa77d"
from {username='slave'} at 127.0.0.1:40930
I20240503 08:09:19.169888 32686 raft_consensus.cc:482] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:19.169907 32686 raft_consensus.cc:3037] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:19.170742 32686 raft_consensus.cc:504] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:19.170954 32686 leader_election.cc:288] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [CANDIDATE]: Term 2 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:19.171113 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c3a314149b4921b6cddce536a010e9" candidate_uuid: "15ce703959504369884c5eac0c7fa77d" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:19.171123 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c3a314149b4921b6cddce536a010e9" candidate_uuid: "15ce703959504369884c5eac0c7fa77d" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:19.171212 32362 raft_consensus.cc:3032] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:19.171226 32436 raft_consensus.cc:3037] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:19.171229 32362 raft_consensus.cc:729] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:19.171308 32362 consensus_queue.cc:260] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:19.171399 32362 raft_consensus.cc:3037] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:19.171875 32436 raft_consensus.cc:2454] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ce703959504369884c5eac0c7fa77d in term 2.
I20240503 08:09:19.171938 32362 raft_consensus.cc:2454] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ce703959504369884c5eac0c7fa77d in term 2.
I20240503 08:09:19.172065 32650 leader_election.cc:302] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ce703959504369884c5eac0c7fa77d, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:19.172247 523 raft_consensus.cc:2781] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:19.172360 523 raft_consensus.cc:686] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 LEADER]: Becoming Leader. State: Replica: 15ce703959504369884c5eac0c7fa77d, State: Running, Role: LEADER
I20240503 08:09:19.172436 523 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:19.172925 32181 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d reported cstate change: term changed from 1 to 2, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to 15ce703959504369884c5eac0c7fa77d (127.28.55.68). New cstate: current_term: 2 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:19.202948 32283 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:19.203231 470 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:19.204365 32436 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:19.204578 470 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:19.264827 32362 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:19.265108 515 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:19.266103 32283 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:19.266270 515 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:19.319486 32283 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240503 08:09:19.319774 470 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:19.321455 32436 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 3 index: 26. (index mismatch)
I20240503 08:09:19.321717 470 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:19.369902 32362 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:19.370445 32283 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 08:09:19.370468 32436 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
W20240503 08:09:19.370571 32327 consensus_peers.cc:485] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 -> Peer 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733): Couldn't send request to peer 08561681ad9a4dc5be7c769e0a959ca9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2edb5018f69946fdb6ae4a792362ae97. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:19.370702 470 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:19.370939 512 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:19.371572 512 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Committing config change with OpId 3.27: config changed from index -1 to 27, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:19.371871 32283 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Committing config change with OpId 3.27: config changed from index -1 to 27, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:19.372637 32436 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Committing config change with OpId 3.27: config changed from index -1 to 27, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:19.372879 32181 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: config changed from index -1 to 27, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New cstate: current_term: 3 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:19.376406 32283 consensus_queue.cc:237] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 2.24, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:19.376993 32436 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:09:19.377110 32362 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20240503 08:09:19.377306 540 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:19.377421 32613 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 08:09:19.378247 540 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Committing config change with OpId 2.25: config changed from index -1 to 25, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:19.378388 32362 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.25: config changed from index -1 to 25, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:19.378446 32436 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Committing config change with OpId 2.25: config changed from index -1 to 25, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:19.379678 32184 catalog_manager.cc:5484] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: config changed from index -1 to 25, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:19.380224 32362 consensus_queue.cc:237] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
W20240503 08:09:19.380369 32250 consensus_peers.cc:485] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 -> Peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117): Couldn't send request to peer 15ce703959504369884c5eac0c7fa77d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b00c72a8ae02442ca1966c3536976263. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:19.380721 32761 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 08:09:19.380882 535 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:19.380961 32283 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240503 08:09:19.381021 32327 consensus_peers.cc:485] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 -> Peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543): Couldn't send request to peer 2ac031e1b6da412da3f1c277da38d38b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7062165e0e3c47c8b43add7f365ec7f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:19.381099 535 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:19.381695 512 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:19.381806 32761 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:19.381811 32283 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:19.382879 32181 catalog_manager.cc:5484] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 reported cstate change: config changed from index 27 to 28, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New cstate: current_term: 1 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
W20240503 08:09:19.383491 32233 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240503 08:09:19.384580 32233 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 10e0365e2b664690ae51f079c4063b8e, TS 441c21339c5c4941943e09a19ca62fe2 -> TS 2ac031e1b6da412da3f1c277da38d38b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
W20240503 08:09:19.384624 32233 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 10e0365e2b664690ae51f079c4063b8e, TS 441c21339c5c4941943e09a19ca62fe2 -> TS 2ac031e1b6da412da3f1c277da38d38b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
W20240503 08:09:19.386160 32233 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4eeef2123c7341509085d8ca8ff2b2ad, TS 94175e832a7f4428be9adc9cfa9032d9 -> TS 08561681ad9a4dc5be7c769e0a959ca9 move failed, destination replica disappeared from tablet's Raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
W20240503 08:09:19.386207 32233 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 4eeef2123c7341509085d8ca8ff2b2ad, TS 94175e832a7f4428be9adc9cfa9032d9 -> TS 08561681ad9a4dc5be7c769e0a959ca9 move failed, destination replica disappeared from tablet's Raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
W20240503 08:09:19.387259 32233 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet df0b30e22bc04f638c852c02354109b6, TS e4df5ad2486247e89c2e44b31357188e -> TS 15ce703959504369884c5eac0c7fa77d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
W20240503 08:09:19.387301 32233 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet df0b30e22bc04f638c852c02354109b6, TS e4df5ad2486247e89c2e44b31357188e -> TS 15ce703959504369884c5eac0c7fa77d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:19.449255 32436 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:19.449496 32613 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:19.450706 32360 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:19.450920 32613 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:19.532029 32436 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:19.532306 32613 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:19.533434 368 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:19.533737 32613 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:19.565277 32436 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:19.565519 523 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:19.566859 32362 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:19.567152 524 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:19.858507 556 ts_tablet_manager.cc:902] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: Initiating tablet copy from peer 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:19.858971 556 tablet_copy_client.cc:308] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:45189
I20240503 08:09:19.859202 32373 tablet_copy_service.cc:140] P 441c21339c5c4941943e09a19ca62fe2: Received BeginTabletCopySession request for tablet 7062165e0e3c47c8b43add7f365ec7f4 from peer 2ac031e1b6da412da3f1c277da38d38b ({username='slave'} at 127.0.0.1:50282)
I20240503 08:09:19.859295 32373 tablet_copy_service.cc:161] P 441c21339c5c4941943e09a19ca62fe2: Beginning new tablet copy session on tablet 7062165e0e3c47c8b43add7f365ec7f4 from peer 2ac031e1b6da412da3f1c277da38d38b at {username='slave'} at 127.0.0.1:50282: session id = 2ac031e1b6da412da3f1c277da38d38b-7062165e0e3c47c8b43add7f365ec7f4
I20240503 08:09:19.859823 32373 tablet_copy_source_session.cc:205] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:19.860235 556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7062165e0e3c47c8b43add7f365ec7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:19.861768 556 tablet_copy_client.cc:783] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:19.861888 556 tablet_copy_client.cc:647] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:19.862414 560 tablet_copy_client.cc:1002] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:19.891355 561 ts_tablet_manager.cc:902] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: Initiating tablet copy from peer 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:19.891728 561 tablet_copy_client.cc:308] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:45189
I20240503 08:09:19.891927 32373 tablet_copy_service.cc:140] P 441c21339c5c4941943e09a19ca62fe2: Received BeginTabletCopySession request for tablet 2edb5018f69946fdb6ae4a792362ae97 from peer 08561681ad9a4dc5be7c769e0a959ca9 ({username='slave'} at 127.0.0.1:50280)
I20240503 08:09:19.891979 32373 tablet_copy_service.cc:161] P 441c21339c5c4941943e09a19ca62fe2: Beginning new tablet copy session on tablet 2edb5018f69946fdb6ae4a792362ae97 from peer 08561681ad9a4dc5be7c769e0a959ca9 at {username='slave'} at 127.0.0.1:50280: session id = 08561681ad9a4dc5be7c769e0a959ca9-2edb5018f69946fdb6ae4a792362ae97
I20240503 08:09:19.892503 32373 tablet_copy_source_session.cc:205] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:19.892822 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edb5018f69946fdb6ae4a792362ae97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:19.893908 561 tablet_copy_client.cc:783] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:19.894004 561 tablet_copy_client.cc:647] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:19.894510 565 tablet_copy_client.cc:1002] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:19.919804 566 ts_tablet_manager.cc:902] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: Initiating tablet copy from peer 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181)
I20240503 08:09:19.920207 566 tablet_copy_client.cc:308] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.65:46181
I20240503 08:09:19.923535 32297 tablet_copy_service.cc:140] P 94175e832a7f4428be9adc9cfa9032d9: Received BeginTabletCopySession request for tablet b00c72a8ae02442ca1966c3536976263 from peer 15ce703959504369884c5eac0c7fa77d ({username='slave'} at 127.0.0.1:57528)
I20240503 08:09:19.923625 32297 tablet_copy_service.cc:161] P 94175e832a7f4428be9adc9cfa9032d9: Beginning new tablet copy session on tablet b00c72a8ae02442ca1966c3536976263 from peer 15ce703959504369884c5eac0c7fa77d at {username='slave'} at 127.0.0.1:57528: session id = 15ce703959504369884c5eac0c7fa77d-b00c72a8ae02442ca1966c3536976263
I20240503 08:09:19.924176 32297 tablet_copy_source_session.cc:205] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:19.924489 566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00c72a8ae02442ca1966c3536976263. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:19.925774 566 tablet_copy_client.cc:783] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:19.925884 566 tablet_copy_client.cc:647] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:19.926590 570 tablet_copy_client.cc:1002] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:20.671881 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:20.672353 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:20.672447 32283 raft_consensus.cc:595] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:20.672910 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4979660e62114184b95e7c2b04a26584"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:20.672986 32283 raft_consensus.cc:595] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:20.673353 32362 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7062165e0e3c47c8b43add7f365ec7f4"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "08561681ad9a4dc5be7c769e0a959ca9"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:20.673418 32362 raft_consensus.cc:595] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Received request to transfer leadership to TS 08561681ad9a4dc5be7c769e0a959ca9
I20240503 08:09:20.673694 32362 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:20.673751 32362 raft_consensus.cc:595] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Received request to transfer leadership to TS e4df5ad2486247e89c2e44b31357188e
I20240503 08:09:20.673866 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240503 08:09:20.673941 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:20.894722 540 raft_consensus.cc:979] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:20.894843 511 raft_consensus.cc:1067] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:20.895030 32362 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50240
I20240503 08:09:20.895140 32362 raft_consensus.cc:482] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:20.895197 32362 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:20.896051 32362 raft_consensus.cc:504] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:20.896185 32362 leader_election.cc:288] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 3 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:20.896312 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:20.896328 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:20.896378 32283 raft_consensus.cc:3032] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:09:20.896420 32436 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:20.896426 32283 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:20.896493 32283 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:20.896582 32283 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:20.897122 32283 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 3.
I20240503 08:09:20.897123 32436 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 3.
I20240503 08:09:20.897310 32327 leader_election.cc:302] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:20.897423 576 raft_consensus.cc:2781] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:09:20.897478 576 raft_consensus.cc:686] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:20.897555 576 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:20.898100 32180 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 2 to 3, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 3 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:20.959643 535 raft_consensus.cc:979] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower 08561681ad9a4dc5be7c769e0a959ca9 to start an election
I20240503 08:09:20.959765 535 raft_consensus.cc:1067] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Signalling peer 08561681ad9a4dc5be7c769e0a959ca9 to start an election
I20240503 08:09:20.959908 32761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7062165e0e3c47c8b43add7f365ec7f4"
dest_uuid: "08561681ad9a4dc5be7c769e0a959ca9"
from {username='slave'} at 127.0.0.1:37730
I20240503 08:09:20.959992 32761 raft_consensus.cc:482] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:20.960017 32761 raft_consensus.cc:3037] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:20.960685 32761 raft_consensus.cc:504] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:20.960870 32761 leader_election.cc:288] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [CANDIDATE]: Term 2 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:20.961268 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7062165e0e3c47c8b43add7f365ec7f4" candidate_uuid: "08561681ad9a4dc5be7c769e0a959ca9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:20.961344 32360 raft_consensus.cc:3032] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:20.961366 32360 raft_consensus.cc:729] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:20.961410 32360 consensus_queue.cc:260] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:20.961508 32360 raft_consensus.cc:3037] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:20.962297 32360 raft_consensus.cc:2454] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08561681ad9a4dc5be7c769e0a959ca9 in term 2.
I20240503 08:09:20.962596 32724 leader_election.cc:302] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08561681ad9a4dc5be7c769e0a959ca9, 441c21339c5c4941943e09a19ca62fe2; no voters:
I20240503 08:09:20.962836 584 raft_consensus.cc:2781] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:20.962980 584 raft_consensus.cc:686] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEADER]: Becoming Leader. State: Replica: 08561681ad9a4dc5be7c769e0a959ca9, State: Running, Role: LEADER
I20240503 08:09:20.963079 584 consensus_queue.cc:237] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:20.963639 32180 catalog_manager.cc:5484] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 reported cstate change: term changed from 1 to 2, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69). New cstate: current_term: 2 leader_uuid: "08561681ad9a4dc5be7c769e0a959ca9" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:20.964542 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7062165e0e3c47c8b43add7f365ec7f4" candidate_uuid: "08561681ad9a4dc5be7c769e0a959ca9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:20.964617 32283 raft_consensus.cc:3037] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:20.965292 32283 raft_consensus.cc:2454] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08561681ad9a4dc5be7c769e0a959ca9 in term 2.
I20240503 08:09:21.149459 540 raft_consensus.cc:979] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:21.149597 540 raft_consensus.cc:1067] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:21.149770 32360 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4979660e62114184b95e7c2b04a26584"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50240
I20240503 08:09:21.149863 32360 raft_consensus.cc:482] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:21.149884 32360 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:21.150571 32360 raft_consensus.cc:504] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:21.150705 32360 leader_election.cc:288] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 3 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:21.150910 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:21.150975 32283 raft_consensus.cc:3032] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:09:21.150995 32283 raft_consensus.cc:729] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:21.151041 32283 consensus_queue.cc:260] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:21.151125 32283 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:21.151082 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:21.151190 32436 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:21.151742 32283 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 3.
I20240503 08:09:21.151805 32436 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 3.
I20240503 08:09:21.151986 32328 leader_election.cc:302] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:21.152083 535 raft_consensus.cc:2781] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:09:21.152135 535 raft_consensus.cc:686] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:21.152184 535 consensus_queue.cc:237] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:21.152889 32180 catalog_manager.cc:5484] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 2 to 3, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 3 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:21.179685 575 raft_consensus.cc:979] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:21.179790 575 raft_consensus.cc:1067] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Signalling peer e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:21.179977 32436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:57356
I20240503 08:09:21.180058 32436 raft_consensus.cc:482] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:21.180083 32436 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:21.180985 32436 raft_consensus.cc:504] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:21.181141 32436 leader_election.cc:288] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 3 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:21.181242 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:21.181304 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:21.181337 32283 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:21.181425 32360 raft_consensus.cc:3032] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:09:21.181453 32360 raft_consensus.cc:729] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:21.181510 32360 consensus_queue.cc:260] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:21.181607 32360 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:21.182132 32283 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 3.
I20240503 08:09:21.182255 32360 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 3.
I20240503 08:09:21.182339 32403 leader_election.cc:302] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:21.182441 489 raft_consensus.cc:2781] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:09:21.182606 489 raft_consensus.cc:686] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 3 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:21.182665 489 consensus_queue.cc:237] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:21.183265 32180 catalog_manager.cc:5484] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 2 to 3, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 3 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:21.426798 32283 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Refusing update from remote peer 08561681ad9a4dc5be7c769e0a959ca9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:21.427042 584 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:21.428305 32360 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 08561681ad9a4dc5be7c769e0a959ca9: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:21.428596 594 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240503 08:09:21.432085 32725 consensus_peers.cc:485] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 -> Peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543): Couldn't send request to peer 2ac031e1b6da412da3f1c277da38d38b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:21.476275 32283 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 08:09:21.476598 576 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:21.477986 32436 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 08:09:21.478207 576 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:21.568837 32283 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 08:09:21.569065 576 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:21.570264 32436 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 08:09:21.570598 575 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:21.646399 32360 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 08:09:21.646626 489 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:21.647907 32281 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 08:09:21.648136 605 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:21.862715 556 tablet_copy_client.cc:523] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:21.863749 556 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap starting.
I20240503 08:09:21.867681 556 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:21.867904 556 tablet_bootstrap.cc:492] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap complete.
I20240503 08:09:21.868033 556 ts_tablet_manager.cc:1367] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 08:09:21.868216 556 raft_consensus.cc:348] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:21.868311 556 raft_consensus.cc:729] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ac031e1b6da412da3f1c277da38d38b, State: Initialized, Role: LEARNER
I20240503 08:09:21.868345 556 consensus_queue.cc:260] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:21.868499 556 ts_tablet_manager.cc:1398] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:09:21.868716 32373 tablet_copy_service.cc:342] P 441c21339c5c4941943e09a19ca62fe2: Request end of tablet copy session 2ac031e1b6da412da3f1c277da38d38b-7062165e0e3c47c8b43add7f365ec7f4 received from {username='slave'} at 127.0.0.1:50282
I20240503 08:09:21.868772 32373 tablet_copy_service.cc:433] P 441c21339c5c4941943e09a19ca62fe2: ending tablet copy session 2ac031e1b6da412da3f1c277da38d38b-7062165e0e3c47c8b43add7f365ec7f4 on tablet 7062165e0e3c47c8b43add7f365ec7f4 with peer 2ac031e1b6da412da3f1c277da38d38b
I20240503 08:09:21.894742 561 tablet_copy_client.cc:523] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:21.895998 561 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap starting.
I20240503 08:09:21.899914 561 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:21.900203 561 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap complete.
I20240503 08:09:21.900367 561 ts_tablet_manager.cc:1367] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20240503 08:09:21.900580 561 raft_consensus.cc:348] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:21.900674 561 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 08561681ad9a4dc5be7c769e0a959ca9, State: Initialized, Role: LEARNER
I20240503 08:09:21.900707 561 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:21.900856 561 ts_tablet_manager.cc:1398] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:21.901075 32373 tablet_copy_service.cc:342] P 441c21339c5c4941943e09a19ca62fe2: Request end of tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-2edb5018f69946fdb6ae4a792362ae97 received from {username='slave'} at 127.0.0.1:50280
I20240503 08:09:21.901152 32373 tablet_copy_service.cc:433] P 441c21339c5c4941943e09a19ca62fe2: ending tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-2edb5018f69946fdb6ae4a792362ae97 on tablet 2edb5018f69946fdb6ae4a792362ae97 with peer 08561681ad9a4dc5be7c769e0a959ca9
I20240503 08:09:21.926909 566 tablet_copy_client.cc:523] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:21.928021 566 tablet_bootstrap.cc:492] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: Bootstrap starting.
I20240503 08:09:21.931607 566 tablet_bootstrap.cc:492] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=84 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:21.931857 566 tablet_bootstrap.cc:492] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: Bootstrap complete.
I20240503 08:09:21.931974 566 ts_tablet_manager.cc:1367] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20240503 08:09:21.932138 566 raft_consensus.cc:348] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:21.932219 566 raft_consensus.cc:729] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 15ce703959504369884c5eac0c7fa77d, State: Initialized, Role: LEARNER
I20240503 08:09:21.932257 566 consensus_queue.cc:260] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:21.932356 566 ts_tablet_manager.cc:1398] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:21.932567 32297 tablet_copy_service.cc:342] P 94175e832a7f4428be9adc9cfa9032d9: Request end of tablet copy session 15ce703959504369884c5eac0c7fa77d-b00c72a8ae02442ca1966c3536976263 received from {username='slave'} at 127.0.0.1:57528
I20240503 08:09:21.932621 32297 tablet_copy_service.cc:433] P 94175e832a7f4428be9adc9cfa9032d9: ending tablet copy session 15ce703959504369884c5eac0c7fa77d-b00c72a8ae02442ca1966c3536976263 on tablet b00c72a8ae02442ca1966c3536976263 with peer 15ce703959504369884c5eac0c7fa77d
I20240503 08:09:21.997260 32760 raft_consensus.cc:1203] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.26->[3.27-3.27] Dedup: 3.27->[]
I20240503 08:09:22.207227 32686 raft_consensus.cc:1203] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.24->[2.25-2.25] Dedup: 2.25->[]
I20240503 08:09:22.295775 368 raft_consensus.cc:3037] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [term 1 LEARNER]: Advancing to term 2
I20240503 08:09:22.602973 613 raft_consensus.cc:1050] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: attempting to promote NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 to VOTER
I20240503 08:09:22.603211 613 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 25, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:22.603709 32281 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:09:22.603822 32436 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:09:22.603832 32760 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEARNER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:09:22.603982 615 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:22.604135 535 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:22.604272 616 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:22.605108 616 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index 27 to 28, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:22.605451 32283 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 27 to 28, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:22.606439 32760 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 27 to 28, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:22.606741 32436 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index 27 to 28, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:22.607523 32184 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: config changed from index 27 to 28, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:22.609099 32362 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 25, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:22.609422 32283 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 08:09:22.609532 32760 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 08:09:22.609582 535 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:22.609687 613 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:22.610139 616 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:22.610266 32283 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:22.611056 32760 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:22.611131 32168 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2edb5018f69946fdb6ae4a792362ae97 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:22.611153 32181 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index 28 to 29, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New cstate: current_term: 3 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:22.612018 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 2edb5018f69946fdb6ae4a792362ae97 with delete_type TABLET_DATA_TOMBSTONED (TS e4df5ad2486247e89c2e44b31357188e not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:57340
I20240503 08:09:22.612236 625 tablet_replica.cc:330] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:22.612287 625 raft_consensus.cc:2229] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:22.612340 625 raft_consensus.cc:2258] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:22.612653 625 ts_tablet_manager.cc:1875] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:22.614128 625 ts_tablet_manager.cc:1888] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.28
I20240503 08:09:22.614188 625 log.cc:1196] T 2edb5018f69946fdb6ae4a792362ae97 P e4df5ad2486247e89c2e44b31357188e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/2edb5018f69946fdb6ae4a792362ae97
I20240503 08:09:22.614495 32169 catalog_manager.cc:4830] TS e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495): tablet 2edb5018f69946fdb6ae4a792362ae97 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:22.674078 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:22.674618 32436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:22.674726 32436 raft_consensus.cc:595] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:22.675046 32362 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:22.675110 32362 raft_consensus.cc:595] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:22.675410 32362 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:22.675472 32362 raft_consensus.cc:595] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:22.675621 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:09:22.675707 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:22.742830 511 raft_consensus.cc:1050] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9: attempting to promote NON_VOTER 15ce703959504369884c5eac0c7fa77d to VOTER
I20240503 08:09:22.743093 511 consensus_queue.cc:237] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 23, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:22.743683 32436 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:22.743719 32362 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:22.743937 32686 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 2 LEARNER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:22.743933 629 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:22.744151 629 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:22.744304 628 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:22.745054 626 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:22.745187 32436 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:22.745230 32362 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:22.746348 32180 catalog_manager.cc:5484] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:22.746836 32686 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index 25 to 26, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:22.747406 32283 consensus_queue.cc:237] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:22.747856 32686 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:09:22.748078 32436 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 08:09:22.748075 626 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:22.748438 628 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:22.748889 629 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index 26 to 27, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:22.749063 32436 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:22.749066 32686 raft_consensus.cc:2932] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:22.750002 32170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b00c72a8ae02442ca1966c3536976263 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:22.749963 32182 catalog_manager.cc:5484] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 26 to 27, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:22.750839 32352 tablet_service.cc:1514] Processing DeleteTablet for tablet b00c72a8ae02442ca1966c3536976263 with delete_type TABLET_DATA_TOMBSTONED (TS 441c21339c5c4941943e09a19ca62fe2 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:22.751073 633 tablet_replica.cc:330] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:22.751147 633 raft_consensus.cc:2229] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:22.751260 633 raft_consensus.cc:2258] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:22.751505 633 ts_tablet_manager.cc:1875] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:22.753273 633 ts_tablet_manager.cc:1888] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.26
I20240503 08:09:22.753335 633 log.cc:1196] T b00c72a8ae02442ca1966c3536976263 P 441c21339c5c4941943e09a19ca62fe2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/b00c72a8ae02442ca1966c3536976263
I20240503 08:09:22.753674 32168 catalog_manager.cc:4830] TS 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189): tablet b00c72a8ae02442ca1966c3536976263 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:22.814783 590 raft_consensus.cc:979] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:22.814910 590 raft_consensus.cc:1067] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:22.815064 32362 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50236
I20240503 08:09:22.815142 32362 raft_consensus.cc:482] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:22.815176 32362 raft_consensus.cc:3037] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:22.816051 32362 raft_consensus.cc:504] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:22.816200 32362 leader_election.cc:288] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 2 election: Requested vote from peers e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495), 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733)
I20240503 08:09:22.816376 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:22.816395 32760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "08561681ad9a4dc5be7c769e0a959ca9"
I20240503 08:09:22.816466 32436 raft_consensus.cc:3032] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:22.816473 32760 raft_consensus.cc:3037] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:22.816504 32436 raft_consensus.cc:729] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:22.816553 32436 consensus_queue.cc:260] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:22.816637 32436 raft_consensus.cc:3037] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:22.817276 32436 raft_consensus.cc:2454] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 2.
I20240503 08:09:22.817296 32760 raft_consensus.cc:2454] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 2.
I20240503 08:09:22.817417 32327 leader_election.cc:302] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:22.817533 620 raft_consensus.cc:2781] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:22.817590 620 raft_consensus.cc:686] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:22.817636 620 consensus_queue.cc:237] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:22.818260 32180 catalog_manager.cc:5484] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 1 to 2, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 2 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:22.857405 636 raft_consensus.cc:1050] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: attempting to promote NON_VOTER 2ac031e1b6da412da3f1c277da38d38b to VOTER
I20240503 08:09:22.857723 636 consensus_queue.cc:237] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:22.858309 32362 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 08561681ad9a4dc5be7c769e0a959ca9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:22.858332 368 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEARNER]: Refusing update from remote peer 08561681ad9a4dc5be7c769e0a959ca9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:22.858489 32283 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Refusing update from remote peer 08561681ad9a4dc5be7c769e0a959ca9: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:22.858613 610 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:22.858752 638 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:22.858947 636 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:22.859995 638 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:22.860044 32362 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:22.860419 32283 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:22.861353 32180 catalog_manager.cc:5484] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 reported cstate change: config changed from index 28 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "08561681ad9a4dc5be7c769e0a959ca9" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:22.861444 368 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:22.862843 32761 consensus_queue.cc:237] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:22.863265 368 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Refusing update from remote peer 08561681ad9a4dc5be7c769e0a959ca9: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:22.863276 32362 raft_consensus.cc:1261] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 08561681ad9a4dc5be7c769e0a959ca9: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:22.863467 638 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:22.863598 641 consensus_queue.cc:1035] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:22.864012 638 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:22.864117 368 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:22.864862 32229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7062165e0e3c47c8b43add7f365ec7f4 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:22.864949 32362 raft_consensus.cc:2932] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:22.865772 32181 catalog_manager.cc:5484] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b reported cstate change: config changed from index 30 to 31, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New cstate: current_term: 2 leader_uuid: "08561681ad9a4dc5be7c769e0a959ca9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:22.866554 32270 tablet_service.cc:1514] Processing DeleteTablet for tablet 7062165e0e3c47c8b43add7f365ec7f4 with delete_type TABLET_DATA_TOMBSTONED (TS 94175e832a7f4428be9adc9cfa9032d9 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:49090
I20240503 08:09:22.866794 642 tablet_replica.cc:330] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:22.866897 642 raft_consensus.cc:2229] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:22.867002 642 raft_consensus.cc:2258] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:22.867224 642 ts_tablet_manager.cc:1875] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:22.868458 642 ts_tablet_manager.cc:1888] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240503 08:09:22.868508 642 log.cc:1196] T 7062165e0e3c47c8b43add7f365ec7f4 P 94175e832a7f4428be9adc9cfa9032d9: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/7062165e0e3c47c8b43add7f365ec7f4
I20240503 08:09:22.868805 32170 catalog_manager.cc:4830] TS 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181): tablet 7062165e0e3c47c8b43add7f365ec7f4 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:22.894271 620 raft_consensus.cc:979] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:22.894368 620 raft_consensus.cc:1067] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:22.894537 32283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49120
I20240503 08:09:22.894631 32283 raft_consensus.cc:482] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:22.894654 32283 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:22.895375 32283 raft_consensus.cc:504] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:22.895532 32283 leader_election.cc:288] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 4 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:22.895705 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:22.895785 32362 raft_consensus.cc:3032] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:09:22.895802 32362 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:22.895709 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:22.895859 32436 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:22.895849 32362 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:22.895943 32362 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:22.896504 32436 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 4.
I20240503 08:09:22.896502 32362 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 4.
I20240503 08:09:22.896714 32250 leader_election.cc:302] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:22.896827 626 raft_consensus.cc:2781] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:09:22.896879 626 raft_consensus.cc:686] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:22.896929 626 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:22.897531 32180 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 3 to 4, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 4 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:23.166734 620 raft_consensus.cc:979] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:23.166867 620 raft_consensus.cc:1067] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:23.167079 32283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49120
I20240503 08:09:23.167176 32283 raft_consensus.cc:482] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:23.167205 32283 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:23.167927 32283 raft_consensus.cc:504] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:23.168102 32283 leader_election.cc:288] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 4 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733)
I20240503 08:09:23.168336 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:23.168424 32362 raft_consensus.cc:3032] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:09:23.168457 32362 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:23.168548 32362 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:23.168664 32362 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:23.169432 32362 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 4.
I20240503 08:09:23.169646 32250 leader_election.cc:302] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:23.169747 626 raft_consensus.cc:2781] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:09:23.169800 626 raft_consensus.cc:686] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:23.169847 626 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:23.170382 32180 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 3 to 4, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 4 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:23.171247 32760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "08561681ad9a4dc5be7c769e0a959ca9"
I20240503 08:09:23.171324 32760 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:23.171906 32760 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 4.
I20240503 08:09:23.287173 32760 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:09:23.287462 620 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:23.288774 32436 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:09:23.288964 535 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:23.374194 32436 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240503 08:09:23.374450 626 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:23.375687 32362 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240503 08:09:23.375876 626 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:23.618559 32760 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:09:23.618875 511 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:23.620240 32362 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 08:09:23.620537 511 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:23.873222 32686 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:23.874030 32362 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:23.874109 32436 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:23.874313 662 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:23.874450 661 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:23.875283 573 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:23.876442 32436 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:23.876672 32181 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 28 to 30, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New cstate: current_term: 2 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:23.876724 32283 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:23.877117 32360 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 08:09:23.877202 32434 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
W20240503 08:09:23.877296 32251 consensus_peers.cc:485] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 -> Peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543): Couldn't send request to peer 2ac031e1b6da412da3f1c277da38d38b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10e0365e2b664690ae51f079c4063b8e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:23.877336 626 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:23.877465 32362 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:23.877525 657 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240503 08:09:23.878183 32650 consensus_peers.cc:485] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d -> Peer 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733): Couldn't send request to peer 08561681ad9a4dc5be7c769e0a959ca9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55c3a314149b4921b6cddce536a010e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:23.878496 626 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:23.878675 32436 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:23.878882 32360 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Committing config change with OpId 4.29: config changed from index -1 to 29, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:23.879925 32184 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: config changed from index -1 to 29, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New cstate: current_term: 4 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:23.880597 32360 consensus_queue.cc:237] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:23.881150 32436 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:09:23.881165 32283 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
W20240503 08:09:23.881297 32326 consensus_peers.cc:485] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 -> Peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117): Couldn't send request to peer 15ce703959504369884c5eac0c7fa77d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4979660e62114184b95e7c2b04a26584. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:23.881429 656 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:23.881527 535 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:23.882273 535 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:23.882404 32436 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:23.882412 32283 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:23.883468 32182 catalog_manager.cc:5484] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index -1 to 28, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New cstate: current_term: 3 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:23.883677 32283 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:23.884171 32360 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
W20240503 08:09:23.884297 32250 consensus_peers.cc:485] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 -> Peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117): Couldn't send request to peer 15ce703959504369884c5eac0c7fa77d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2edb5018f69946fdb6ae4a792362ae97. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:23.884397 511 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:23.884582 32760 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:09:23.884832 626 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:23.884976 626 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 29 to 31, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:23.885121 32360 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 29 to 31, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:23.885493 32760 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 29 to 31, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:23.886277 32180 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: config changed from index 29 to 31, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New cstate: current_term: 4 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:23.889353 368 consensus_queue.cc:237] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:23.889936 32360 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:23.889936 32283 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:23.890163 677 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:23.890298 676 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:23.891104 555 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:23.891182 32360 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:23.892302 32283 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:23.892393 32182 catalog_manager.cc:5484] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b reported cstate change: config changed from index 28 to 30, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New cstate: current_term: 2 leader_uuid: "2ac031e1b6da412da3f1c277da38d38b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:09:23.892652 332 consensus_peers.cc:485] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b -> Peer 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733): Couldn't send request to peer 08561681ad9a4dc5be7c769e0a959ca9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ac923b6f4774b589c220467ccb7fd8f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:23.893469 32360 consensus_queue.cc:237] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:23.893893 32760 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:23.894003 32436 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:23.894061 620 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240503 08:09:23.894275 32327 consensus_peers.cc:485] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 -> Peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543): Couldn't send request to peer 2ac031e1b6da412da3f1c277da38d38b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07d3a63eb2f640bc9f1ea96a6e071bd5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:23.894282 620 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:23.894616 656 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:23.894748 32760 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:23.895859 32184 catalog_manager.cc:5484] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 reported cstate change: config changed from index 27 to 29, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New cstate: current_term: 2 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:23.896248 32436 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:24.313338 683 ts_tablet_manager.cc:902] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: Initiating tablet copy from peer 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181)
I20240503 08:09:24.313755 683 tablet_copy_client.cc:308] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.65:46181
I20240503 08:09:24.313969 32297 tablet_copy_service.cc:140] P 94175e832a7f4428be9adc9cfa9032d9: Received BeginTabletCopySession request for tablet 10e0365e2b664690ae51f079c4063b8e from peer 2ac031e1b6da412da3f1c277da38d38b ({username='slave'} at 127.0.0.1:57510)
I20240503 08:09:24.314049 32297 tablet_copy_service.cc:161] P 94175e832a7f4428be9adc9cfa9032d9: Beginning new tablet copy session on tablet 10e0365e2b664690ae51f079c4063b8e from peer 2ac031e1b6da412da3f1c277da38d38b at {username='slave'} at 127.0.0.1:57510: session id = 2ac031e1b6da412da3f1c277da38d38b-10e0365e2b664690ae51f079c4063b8e
I20240503 08:09:24.314723 32297 tablet_copy_source_session.cc:205] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:24.315143 683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e0365e2b664690ae51f079c4063b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:24.316798 683 tablet_copy_client.cc:783] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:24.316956 683 tablet_copy_client.cc:647] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:24.317504 687 tablet_copy_client.cc:1002] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:24.340674 688 ts_tablet_manager.cc:902] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: Initiating tablet copy from peer 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:24.341115 688 tablet_copy_client.cc:308] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:45189
I20240503 08:09:24.341405 32373 tablet_copy_service.cc:140] P 441c21339c5c4941943e09a19ca62fe2: Received BeginTabletCopySession request for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 from peer 2ac031e1b6da412da3f1c277da38d38b ({username='slave'} at 127.0.0.1:50282)
I20240503 08:09:24.341477 32373 tablet_copy_service.cc:161] P 441c21339c5c4941943e09a19ca62fe2: Beginning new tablet copy session on tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 from peer 2ac031e1b6da412da3f1c277da38d38b at {username='slave'} at 127.0.0.1:50282: session id = 2ac031e1b6da412da3f1c277da38d38b-07d3a63eb2f640bc9f1ea96a6e071bd5
I20240503 08:09:24.342105 32373 tablet_copy_source_session.cc:205] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:24.342491 688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:24.344612 688 tablet_copy_client.cc:783] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:24.344785 688 tablet_copy_client.cc:647] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:24.345332 692 tablet_copy_client.cc:1002] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:24.360116 693 ts_tablet_manager.cc:902] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: Initiating tablet copy from peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117)
I20240503 08:09:24.360491 693 tablet_copy_client.cc:308] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.68:43117
I20240503 08:09:24.364022 32696 tablet_copy_service.cc:140] P 15ce703959504369884c5eac0c7fa77d: Received BeginTabletCopySession request for tablet 55c3a314149b4921b6cddce536a010e9 from peer 08561681ad9a4dc5be7c769e0a959ca9 ({username='slave'} at 127.0.0.1:57978)
I20240503 08:09:24.364116 32696 tablet_copy_service.cc:161] P 15ce703959504369884c5eac0c7fa77d: Beginning new tablet copy session on tablet 55c3a314149b4921b6cddce536a010e9 from peer 08561681ad9a4dc5be7c769e0a959ca9 at {username='slave'} at 127.0.0.1:57978: session id = 08561681ad9a4dc5be7c769e0a959ca9-55c3a314149b4921b6cddce536a010e9
I20240503 08:09:24.364787 32696 tablet_copy_source_session.cc:205] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:24.365171 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c3a314149b4921b6cddce536a010e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:24.366611 693 tablet_copy_client.cc:783] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:24.366730 693 tablet_copy_client.cc:647] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:24.367290 697 tablet_copy_client.cc:1002] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:24.384063 699 ts_tablet_manager.cc:902] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: Initiating tablet copy from peer 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:24.384430 699 tablet_copy_client.cc:308] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:45189
I20240503 08:09:24.384608 32373 tablet_copy_service.cc:140] P 441c21339c5c4941943e09a19ca62fe2: Received BeginTabletCopySession request for tablet 4979660e62114184b95e7c2b04a26584 from peer 15ce703959504369884c5eac0c7fa77d ({username='slave'} at 127.0.0.1:50294)
I20240503 08:09:24.384680 32373 tablet_copy_service.cc:161] P 441c21339c5c4941943e09a19ca62fe2: Beginning new tablet copy session on tablet 4979660e62114184b95e7c2b04a26584 from peer 15ce703959504369884c5eac0c7fa77d at {username='slave'} at 127.0.0.1:50294: session id = 15ce703959504369884c5eac0c7fa77d-4979660e62114184b95e7c2b04a26584
I20240503 08:09:24.385334 32373 tablet_copy_source_session.cc:205] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:24.385672 699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4979660e62114184b95e7c2b04a26584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:24.387068 699 tablet_copy_client.cc:783] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:24.387219 699 tablet_copy_client.cc:647] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:24.387796 703 tablet_copy_client.cc:1002] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:24.418385 704 ts_tablet_manager.cc:902] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: Initiating tablet copy from peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543)
I20240503 08:09:24.418745 704 tablet_copy_client.cc:308] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.70:33543
I20240503 08:09:24.418963 378 tablet_copy_service.cc:140] P 2ac031e1b6da412da3f1c277da38d38b: Received BeginTabletCopySession request for tablet 7ac923b6f4774b589c220467ccb7fd8f from peer 08561681ad9a4dc5be7c769e0a959ca9 ({username='slave'} at 127.0.0.1:53886)
I20240503 08:09:24.419026 378 tablet_copy_service.cc:161] P 2ac031e1b6da412da3f1c277da38d38b: Beginning new tablet copy session on tablet 7ac923b6f4774b589c220467ccb7fd8f from peer 08561681ad9a4dc5be7c769e0a959ca9 at {username='slave'} at 127.0.0.1:53886: session id = 08561681ad9a4dc5be7c769e0a959ca9-7ac923b6f4774b589c220467ccb7fd8f
I20240503 08:09:24.419704 378 tablet_copy_source_session.cc:205] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:24.420030 704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac923b6f4774b589c220467ccb7fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:24.421629 704 tablet_copy_client.cc:783] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:24.421787 704 tablet_copy_client.cc:647] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:24.422324 707 tablet_copy_client.cc:1002] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:24.472730 709 ts_tablet_manager.cc:902] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: Initiating tablet copy from peer 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181)
I20240503 08:09:24.473165 709 tablet_copy_client.cc:308] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.65:46181
I20240503 08:09:24.473376 32297 tablet_copy_service.cc:140] P 94175e832a7f4428be9adc9cfa9032d9: Received BeginTabletCopySession request for tablet 2edb5018f69946fdb6ae4a792362ae97 from peer 15ce703959504369884c5eac0c7fa77d ({username='slave'} at 127.0.0.1:57528)
I20240503 08:09:24.473433 32297 tablet_copy_service.cc:161] P 94175e832a7f4428be9adc9cfa9032d9: Beginning new tablet copy session on tablet 2edb5018f69946fdb6ae4a792362ae97 from peer 15ce703959504369884c5eac0c7fa77d at {username='slave'} at 127.0.0.1:57528: session id = 15ce703959504369884c5eac0c7fa77d-2edb5018f69946fdb6ae4a792362ae97
I20240503 08:09:24.474238 32297 tablet_copy_source_session.cc:205] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:24.474582 709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edb5018f69946fdb6ae4a792362ae97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:24.476141 709 tablet_copy_client.cc:783] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:24.476266 709 tablet_copy_client.cc:647] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:24.476828 713 tablet_copy_client.cc:1002] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:24.676044 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:24.676892 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:24.677007 32283 raft_consensus.cc:595] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:24.677495 32360 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:24.677601 32360 raft_consensus.cc:595] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Received request to transfer leadership to TS e4df5ad2486247e89c2e44b31357188e
I20240503 08:09:24.677911 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:24.677986 32283 raft_consensus.cc:595] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:24.678175 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:09:24.678259 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:24.793713 511 raft_consensus.cc:979] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:24.793820 511 raft_consensus.cc:1067] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:24.794034 32360 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50240
I20240503 08:09:24.794108 32360 raft_consensus.cc:482] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:24.794133 32360 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:24.795109 32360 raft_consensus.cc:504] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:24.795331 32360 leader_election.cc:288] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 5 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:24.795425 32281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:24.795444 32436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:24.795485 32281 raft_consensus.cc:3032] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:09:24.795488 32436 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:24.795537 32281 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:24.795588 32281 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:24.795677 32281 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:24.796305 32281 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 5.
I20240503 08:09:24.796305 32436 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 5.
I20240503 08:09:24.796563 32327 leader_election.cc:302] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:24.796677 714 raft_consensus.cc:2781] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:09:24.796736 714 raft_consensus.cc:686] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:24.796797 714 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:24.797320 32184 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 4 to 5, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 5 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:24.800035 32360 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:24.800124 32360 raft_consensus.cc:595] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Received request to transfer leadership
I20240503 08:09:24.802049 32360 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:24.802124 32360 raft_consensus.cc:595] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Received request to transfer leadership
W20240503 08:09:24.802248 32168 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 10e0365e2b664690ae51f079c4063b8e already in progress [suppressed 2 similar messages]
I20240503 08:09:24.802738 32360 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:24.802791 32360 raft_consensus.cc:595] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Received request to transfer leadership
W20240503 08:09:24.802999 32233 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 10e0365e2b664690ae51f079c4063b8e already in progress
W20240503 08:09:24.803043 32233 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 10e0365e2b664690ae51f079c4063b8e already in progress
I20240503 08:09:24.868616 656 raft_consensus.cc:979] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:24.868719 656 raft_consensus.cc:1067] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Signalling peer e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:24.868885 32436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:57356
I20240503 08:09:24.868963 32436 raft_consensus.cc:482] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:24.868988 32436 raft_consensus.cc:3037] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:24.869987 32436 raft_consensus.cc:504] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:24.870164 32436 leader_election.cc:288] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 2 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:24.870285 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:24.870398 32360 raft_consensus.cc:3032] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:24.870285 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:24.870460 32360 raft_consensus.cc:729] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:24.870477 32283 raft_consensus.cc:3037] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:24.870513 32360 consensus_queue.cc:260] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:24.870581 32360 raft_consensus.cc:3037] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:24.871292 32283 raft_consensus.cc:2454] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 2.
I20240503 08:09:24.871505 32403 leader_election.cc:302] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:24.871292 32360 raft_consensus.cc:2454] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 2.
I20240503 08:09:24.871609 634 raft_consensus.cc:2781] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:24.871731 634 raft_consensus.cc:686] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 2 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:24.871779 634 consensus_queue.cc:237] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:24.872344 32184 catalog_manager.cc:5484] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 1 to 2, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 2 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:24.968556 715 raft_consensus.cc:979] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:24.968770 715 raft_consensus.cc:1067] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:24.968982 32360 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50240
I20240503 08:09:24.969063 32360 raft_consensus.cc:482] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:24.969100 32360 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:24.969981 32360 raft_consensus.cc:504] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:24.970151 32360 leader_election.cc:288] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 5 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733)
I20240503 08:09:24.970248 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:24.970311 32283 raft_consensus.cc:3032] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:09:24.970268 32760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "08561681ad9a4dc5be7c769e0a959ca9"
I20240503 08:09:24.970345 32283 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:24.970366 32760 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:24.970412 32283 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:24.970515 32283 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:24.971119 32283 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 5.
I20240503 08:09:24.971127 32760 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 5.
I20240503 08:09:24.971323 32327 leader_election.cc:302] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08561681ad9a4dc5be7c769e0a959ca9, 441c21339c5c4941943e09a19ca62fe2; no voters:
I20240503 08:09:24.971452 722 raft_consensus.cc:2781] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:09:24.971506 722 raft_consensus.cc:686] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:24.971576 722 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:24.972193 32184 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 4 to 5, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 5 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:24.973573 32360 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:24.973639 32360 raft_consensus.cc:595] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Received request to transfer leadership
I20240503 08:09:24.975253 32360 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:24.975330 32360 raft_consensus.cc:595] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Received request to transfer leadership
I20240503 08:09:24.975608 32360 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:24.975659 32360 raft_consensus.cc:595] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Received request to transfer leadership
W20240503 08:09:24.975800 32233 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 2edb5018f69946fdb6ae4a792362ae97 already in progress
W20240503 08:09:24.975844 32233 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 2edb5018f69946fdb6ae4a792362ae97 already in progress
W20240503 08:09:25.180567 32327 consensus_peers.cc:485] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 -> Peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543): Couldn't send request to peer 2ac031e1b6da412da3f1c277da38d38b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:25.331262 32436 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240503 08:09:25.331502 722 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:25.332242 656 raft_consensus.cc:979] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:25.332310 656 raft_consensus.cc:1067] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Signalling peer e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:25.332580 32434 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:57356
I20240503 08:09:25.332656 32434 raft_consensus.cc:482] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:25.332684 32434 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:09:25.333459 32434 raft_consensus.cc:504] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:25.333618 32434 leader_election.cc:288] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 6 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:25.333774 32283 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240503 08:09:25.333748 32281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:25.333936 32281 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:09:25.334152 656 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:25.334745 32281 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 6.
I20240503 08:09:25.334825 32283 raft_consensus.cc:1226] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Rejecting Update request from peer 441c21339c5c4941943e09a19ca62fe2 for earlier term 5. Current term is 6. Ops: [5.30-5.30]
I20240503 08:09:25.333920 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:25.334915 32403 leader_election.cc:302] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:25.334967 32360 raft_consensus.cc:3032] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:09:25.334993 32360 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:25.335018 634 raft_consensus.cc:2781] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:09:25.335069 634 raft_consensus.cc:686] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 6 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:25.335043 32360 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 30, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:25.335100 656 consensus_queue.cc:1046] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: INVALID_TERM, Last received: 4.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:25.335120 634 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 30, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:25.335253 32360 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:09:25.336014 32360 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 6.
I20240503 08:09:25.336179 32184 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 5 to 6, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 6 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:25.385900 32360 raft_consensus.cc:1261] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:25.386142 634 consensus_queue.cc:1035] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:25.387573 32281 raft_consensus.cc:1261] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 08:09:25.387709 32283 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 08:09:25.387882 714 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:25.387882 634 consensus_queue.cc:1035] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 08:09:25.388806 656 raft_consensus.cc:979] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:25.388872 656 raft_consensus.cc:1067] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
W20240503 08:09:25.389199 32326 consensus_peers.cc:485] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 -> Peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117): Couldn't send request to peer 15ce703959504369884c5eac0c7fa77d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:25.389238 32760 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 5 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 08:09:25.389518 32283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49120
I20240503 08:09:25.389585 32283 raft_consensus.cc:482] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:25.389609 32283 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:09:25.391088 32283 raft_consensus.cc:504] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:25.391237 32283 leader_election.cc:288] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 6 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733)
I20240503 08:09:25.391366 722 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:25.391371 32760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "08561681ad9a4dc5be7c769e0a959ca9"
I20240503 08:09:25.391465 32760 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:09:25.391719 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edb5018f69946fdb6ae4a792362ae97" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:25.391789 32360 raft_consensus.cc:3032] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:09:25.391811 32360 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:25.391871 32360 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:25.391968 32360 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:09:25.392395 32760 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 6.
I20240503 08:09:25.392493 32761 raft_consensus.cc:1226] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 6 FOLLOWER]: Rejecting Update request from peer 441c21339c5c4941943e09a19ca62fe2 for earlier term 5. Current term is 6. Ops: [5.32-5.32]
I20240503 08:09:25.392572 32251 leader_election.cc:302] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08561681ad9a4dc5be7c769e0a959ca9, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:25.392601 32360 raft_consensus.cc:2454] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 6.
I20240503 08:09:25.392653 715 raft_consensus.cc:2781] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:09:25.392688 715 raft_consensus.cc:686] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:25.392750 715 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:25.393213 32180 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 5 to 6, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 6 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:25.750754 32362 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20240503 08:09:25.751053 634 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20240503 08:09:25.752264 32402 consensus_peers.cc:485] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e -> Peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543): Couldn't send request to peer 2ac031e1b6da412da3f1c277da38d38b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:25.752382 32283 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20240503 08:09:25.752555 736 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
W20240503 08:09:25.802636 32250 consensus_peers.cc:485] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 -> Peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117): Couldn't send request to peer 15ce703959504369884c5eac0c7fa77d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:25.836609 32360 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20240503 08:09:25.836819 715 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:09:25.837862 32761 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 6 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20240503 08:09:25.838080 738 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:26.317761 683 tablet_copy_client.cc:523] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:26.319365 683 tablet_bootstrap.cc:492] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap starting.
I20240503 08:09:26.323947 683 tablet_bootstrap.cc:492] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=99 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:26.324188 683 tablet_bootstrap.cc:492] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap complete.
I20240503 08:09:26.324314 683 ts_tablet_manager.cc:1367] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:09:26.324537 683 raft_consensus.cc:348] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:26.324651 683 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 2ac031e1b6da412da3f1c277da38d38b, State: Initialized, Role: LEARNER
I20240503 08:09:26.324687 683 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:26.324851 683 ts_tablet_manager.cc:1398] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20240503 08:09:26.325044 32297 tablet_copy_service.cc:342] P 94175e832a7f4428be9adc9cfa9032d9: Request end of tablet copy session 2ac031e1b6da412da3f1c277da38d38b-10e0365e2b664690ae51f079c4063b8e received from {username='slave'} at 127.0.0.1:57510
I20240503 08:09:26.325106 32297 tablet_copy_service.cc:433] P 94175e832a7f4428be9adc9cfa9032d9: ending tablet copy session 2ac031e1b6da412da3f1c277da38d38b-10e0365e2b664690ae51f079c4063b8e on tablet 10e0365e2b664690ae51f079c4063b8e with peer 2ac031e1b6da412da3f1c277da38d38b
I20240503 08:09:26.345567 688 tablet_copy_client.cc:523] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:26.346669 688 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap starting.
I20240503 08:09:26.350553 688 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:26.350782 688 tablet_bootstrap.cc:492] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap complete.
I20240503 08:09:26.350934 688 ts_tablet_manager.cc:1367] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20240503 08:09:26.351117 688 raft_consensus.cc:348] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:26.351212 688 raft_consensus.cc:729] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2ac031e1b6da412da3f1c277da38d38b, State: Initialized, Role: LEARNER
I20240503 08:09:26.351243 688 consensus_queue.cc:260] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:26.351394 688 ts_tablet_manager.cc:1398] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:26.351579 32373 tablet_copy_service.cc:342] P 441c21339c5c4941943e09a19ca62fe2: Request end of tablet copy session 2ac031e1b6da412da3f1c277da38d38b-07d3a63eb2f640bc9f1ea96a6e071bd5 received from {username='slave'} at 127.0.0.1:50282
I20240503 08:09:26.351630 32373 tablet_copy_service.cc:433] P 441c21339c5c4941943e09a19ca62fe2: ending tablet copy session 2ac031e1b6da412da3f1c277da38d38b-07d3a63eb2f640bc9f1ea96a6e071bd5 on tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 with peer 2ac031e1b6da412da3f1c277da38d38b
I20240503 08:09:26.367506 693 tablet_copy_client.cc:523] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:26.368620 693 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap starting.
I20240503 08:09:26.373719 693 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=89 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:26.374059 693 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap complete.
I20240503 08:09:26.374218 693 ts_tablet_manager.cc:1367] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20240503 08:09:26.374521 693 raft_consensus.cc:348] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:26.374686 693 raft_consensus.cc:729] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 08561681ad9a4dc5be7c769e0a959ca9, State: Initialized, Role: LEARNER
I20240503 08:09:26.374737 693 consensus_queue.cc:260] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:26.374879 693 ts_tablet_manager.cc:1398] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:26.375103 32696 tablet_copy_service.cc:342] P 15ce703959504369884c5eac0c7fa77d: Request end of tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-55c3a314149b4921b6cddce536a010e9 received from {username='slave'} at 127.0.0.1:57978
I20240503 08:09:26.375172 32696 tablet_copy_service.cc:433] P 15ce703959504369884c5eac0c7fa77d: ending tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-55c3a314149b4921b6cddce536a010e9 on tablet 55c3a314149b4921b6cddce536a010e9 with peer 08561681ad9a4dc5be7c769e0a959ca9
I20240503 08:09:26.388024 699 tablet_copy_client.cc:523] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:26.389060 699 tablet_bootstrap.cc:492] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: Bootstrap starting.
I20240503 08:09:26.392114 699 tablet_bootstrap.cc:492] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:26.392321 699 tablet_bootstrap.cc:492] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: Bootstrap complete.
I20240503 08:09:26.392436 699 ts_tablet_manager.cc:1367] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:09:26.392596 699 raft_consensus.cc:348] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:26.392693 699 raft_consensus.cc:729] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 15ce703959504369884c5eac0c7fa77d, State: Initialized, Role: LEARNER
I20240503 08:09:26.392725 699 consensus_queue.cc:260] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:26.392839 699 ts_tablet_manager.cc:1398] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20240503 08:09:26.393077 32373 tablet_copy_service.cc:342] P 441c21339c5c4941943e09a19ca62fe2: Request end of tablet copy session 15ce703959504369884c5eac0c7fa77d-4979660e62114184b95e7c2b04a26584 received from {username='slave'} at 127.0.0.1:50294
I20240503 08:09:26.393133 32373 tablet_copy_service.cc:433] P 441c21339c5c4941943e09a19ca62fe2: ending tablet copy session 15ce703959504369884c5eac0c7fa77d-4979660e62114184b95e7c2b04a26584 on tablet 4979660e62114184b95e7c2b04a26584 with peer 15ce703959504369884c5eac0c7fa77d
I20240503 08:09:26.422585 704 tablet_copy_client.cc:523] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:26.423653 704 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap starting.
I20240503 08:09:26.427399 704 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:26.427634 704 tablet_bootstrap.cc:492] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap complete.
I20240503 08:09:26.427755 704 ts_tablet_manager.cc:1367] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:09:26.427955 704 raft_consensus.cc:348] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:26.428050 704 raft_consensus.cc:729] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 08561681ad9a4dc5be7c769e0a959ca9, State: Initialized, Role: LEARNER
I20240503 08:09:26.428081 704 consensus_queue.cc:260] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:26.428198 704 ts_tablet_manager.cc:1398] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:26.428391 378 tablet_copy_service.cc:342] P 2ac031e1b6da412da3f1c277da38d38b: Request end of tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-7ac923b6f4774b589c220467ccb7fd8f received from {username='slave'} at 127.0.0.1:53886
I20240503 08:09:26.428452 378 tablet_copy_service.cc:433] P 2ac031e1b6da412da3f1c277da38d38b: ending tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-7ac923b6f4774b589c220467ccb7fd8f on tablet 7ac923b6f4774b589c220467ccb7fd8f with peer 08561681ad9a4dc5be7c769e0a959ca9
I20240503 08:09:26.453859 32761 raft_consensus.cc:1203] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240503 08:09:26.477080 709 tablet_copy_client.cc:523] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:26.478220 709 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: Bootstrap starting.
I20240503 08:09:26.481521 709 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:26.481714 709 tablet_bootstrap.cc:492] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: Bootstrap complete.
I20240503 08:09:26.481818 709 ts_tablet_manager.cc:1367] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:09:26.482100 709 raft_consensus.cc:348] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:26.482175 709 raft_consensus.cc:729] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 15ce703959504369884c5eac0c7fa77d, State: Initialized, Role: LEARNER
I20240503 08:09:26.482198 709 consensus_queue.cc:260] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:26.482302 709 ts_tablet_manager.cc:1398] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:26.482796 32297 tablet_copy_service.cc:342] P 94175e832a7f4428be9adc9cfa9032d9: Request end of tablet copy session 15ce703959504369884c5eac0c7fa77d-2edb5018f69946fdb6ae4a792362ae97 received from {username='slave'} at 127.0.0.1:57528
I20240503 08:09:26.482888 32297 tablet_copy_service.cc:433] P 94175e832a7f4428be9adc9cfa9032d9: ending tablet copy session 15ce703959504369884c5eac0c7fa77d-2edb5018f69946fdb6ae4a792362ae97 on tablet 2edb5018f69946fdb6ae4a792362ae97 with peer 15ce703959504369884c5eac0c7fa77d
I20240503 08:09:26.565742 367 raft_consensus.cc:1203] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.28->[2.29-2.29] Dedup: 2.29->[]
I20240503 08:09:26.676528 368 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 4 LEARNER]: Advancing to term 6
I20240503 08:09:26.677731 368 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 6 LEARNER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240503 08:09:26.677937 723 consensus_queue.cc:1037] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:26.678385 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:26.678822 723 raft_consensus.cc:1050] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e: attempting to promote NON_VOTER 2ac031e1b6da412da3f1c277da38d38b to VOTER
I20240503 08:09:26.679008 723 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 6.31, Last appended by leader: 30, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:26.679486 368 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 6 LEARNER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20240503 08:09:26.678767 32434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:26.679553 32434 raft_consensus.cc:595] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 6 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:26.679556 32360 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20240503 08:09:26.679745 32283 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20240503 08:09:26.679910 752 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:26.680099 32434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:26.680150 32434 raft_consensus.cc:595] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:26.680022 753 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:26.680456 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "2ac031e1b6da412da3f1c277da38d38b"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:26.680543 32283 raft_consensus.cc:595] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Received request to transfer leadership to TS 2ac031e1b6da412da3f1c277da38d38b
I20240503 08:09:26.680635 754 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:26.680704 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 08:09:26.680765 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:26.681344 32360 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Committing config change with OpId 6.32: config changed from index 29 to 32, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:26.681272 753 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 6 LEADER]: Committing config change with OpId 6.32: config changed from index 29 to 32, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:26.682418 752 raft_consensus.cc:979] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:26.682600 752 raft_consensus.cc:1067] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 6 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:26.682703 32180 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index 29 to 32, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:26.682771 32281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49104
I20240503 08:09:26.682783 32283 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Committing config change with OpId 6.32: config changed from index 29 to 32, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:26.683833 32281 raft_consensus.cc:482] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:26.683884 32281 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:09:26.684579 32281 raft_consensus.cc:504] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:26.684731 32281 leader_election.cc:288] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 7 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495), 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543)
I20240503 08:09:26.684803 32360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 32 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:26.684860 32360 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:09:26.682855 368 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 6 FOLLOWER]: Committing config change with OpId 6.32: config changed from index 29 to 32, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:26.685099 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 32 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:26.685165 32434 raft_consensus.cc:3032] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:09:26.685187 32434 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:26.685245 32434 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 6.32, Last appended by leader: 32, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:26.685317 32434 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:09:26.685492 32360 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 7.
I20240503 08:09:26.685632 367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 7 candidate_status { last_received { term: 6 index: 32 } } ignore_live_leader: true dest_uuid: "2ac031e1b6da412da3f1c277da38d38b"
I20240503 08:09:26.685927 32434 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 7.
I20240503 08:09:26.686041 32251 leader_election.cc:302] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters: 2ac031e1b6da412da3f1c277da38d38b
I20240503 08:09:26.686147 715 raft_consensus.cc:2781] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:09:26.686193 715 raft_consensus.cc:686] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 7 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:26.686244 715 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 6.32, Last appended by leader: 32, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
W20240503 08:09:26.686604 32169 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 10e0365e2b664690ae51f079c4063b8e with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER failed with leader e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e4df5ad2486247e89c2e44b31357188e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:26.686695 753 raft_consensus.cc:979] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:26.686740 753 raft_consensus.cc:1067] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:26.686743 32180 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 6 to 7, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 7 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:26.686899 32281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49104
I20240503 08:09:26.686944 32281 raft_consensus.cc:482] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:26.686978 32281 raft_consensus.cc:3037] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:26.687531 32281 raft_consensus.cc:504] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:26.687639 32281 leader_election.cc:288] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Requested vote from peers e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495), 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117)
I20240503 08:09:26.687748 32686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "15ce703959504369884c5eac0c7fa77d"
I20240503 08:09:26.687806 32686 raft_consensus.cc:3037] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:26.688472 32686 raft_consensus.cc:2454] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:26.688618 32250 leader_election.cc:302] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ce703959504369884c5eac0c7fa77d, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:26.688618 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:26.688709 32434 raft_consensus.cc:3032] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:26.688728 32434 raft_consensus.cc:729] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:26.688745 726 raft_consensus.cc:2781] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:26.688796 726 raft_consensus.cc:686] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:26.688776 32434 consensus_queue.cc:260] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:26.688851 32434 raft_consensus.cc:3037] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:26.688840 726 consensus_queue.cc:237] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:26.689512 32434 raft_consensus.cc:2454] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 2.
I20240503 08:09:26.689503 32180 catalog_manager.cc:5484] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 1 to 2, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:26.757370 32686 raft_consensus.cc:1203] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 3 LEARNER]: Deduplicated request from leader. Original: 3.27->[3.28-3.28] Dedup: 3.28->[]
I20240503 08:09:26.823151 32761 raft_consensus.cc:1203] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240503 08:09:26.963594 32686 raft_consensus.cc:3037] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 4 LEARNER]: Advancing to term 6
I20240503 08:09:26.964560 32686 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 6 LEARNER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 08:09:26.964768 726 consensus_queue.cc:1037] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:26.965718 726 raft_consensus.cc:1050] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: attempting to promote NON_VOTER 15ce703959504369884c5eac0c7fa77d to VOTER
I20240503 08:09:26.965934 726 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 6.33, Last appended by leader: 32, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:26.966467 32356 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240503 08:09:26.966496 32685 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 6 LEARNER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240503 08:09:26.966496 32761 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 6 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20240503 08:09:26.966738 766 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:09:26.966858 726 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:09:26.966996 715 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:09:26.967756 715 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 LEADER]: Committing config change with OpId 6.34: config changed from index 31 to 34, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:26.967931 32761 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 6 FOLLOWER]: Committing config change with OpId 6.34: config changed from index 31 to 34, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:26.969096 32180 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: config changed from index 31 to 34, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:26.969103 32686 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 6 FOLLOWER]: Committing config change with OpId 6.34: config changed from index 31 to 34, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:26.969522 32356 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Committing config change with OpId 6.34: config changed from index 31 to 34, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:26.970602 32281 consensus_queue.cc:237] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 6.34, Last appended by leader: 32, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:26.970952 32686 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 6 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20240503 08:09:26.970952 32761 raft_consensus.cc:1261] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 6 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20240503 08:09:26.971112 766 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:09:26.971218 715 consensus_queue.cc:1035] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:09:26.971727 726 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 LEADER]: Committing config change with OpId 6.35: config changed from index 34 to 35, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:26.971848 32761 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 6 FOLLOWER]: Committing config change with OpId 6.35: config changed from index 34 to 35, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:26.972754 32685 raft_consensus.cc:2932] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 6 FOLLOWER]: Committing config change with OpId 6.35: config changed from index 34 to 35, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:26.972859 32170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2edb5018f69946fdb6ae4a792362ae97 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:26.972880 32184 catalog_manager.cc:5484] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: config changed from index 34 to 35, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New cstate: current_term: 6 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:26.973768 32352 tablet_service.cc:1514] Processing DeleteTablet for tablet 2edb5018f69946fdb6ae4a792362ae97 with delete_type TABLET_DATA_TOMBSTONED (TS 441c21339c5c4941943e09a19ca62fe2 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:26.974043 769 tablet_replica.cc:330] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:26.974117 769 raft_consensus.cc:2229] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:26.974195 769 raft_consensus.cc:2258] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:26.974411 769 ts_tablet_manager.cc:1875] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:26.975852 769 ts_tablet_manager.cc:1888] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.34
I20240503 08:09:26.975935 769 log.cc:1196] T 2edb5018f69946fdb6ae4a792362ae97 P 441c21339c5c4941943e09a19ca62fe2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/2edb5018f69946fdb6ae4a792362ae97
I20240503 08:09:26.976255 32168 catalog_manager.cc:4830] TS 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189): tablet 2edb5018f69946fdb6ae4a792362ae97 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:27.015285 722 raft_consensus.cc:1050] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2: attempting to promote NON_VOTER 2ac031e1b6da412da3f1c277da38d38b to VOTER
I20240503 08:09:27.015530 722 consensus_queue.cc:237] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:27.016217 368 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEARNER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:27.016271 32761 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:27.016271 32434 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 08:09:27.016495 772 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:27.016659 770 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:27.017057 773 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:27.017812 772 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.019083 32434 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.019194 32182 catalog_manager.cc:5484] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index 29 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:27.020439 32761 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.020606 368 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.022248 32360 consensus_queue.cc:237] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:27.022634 367 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:27.022697 32761 raft_consensus.cc:1261] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:27.022902 773 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:27.023021 772 consensus_queue.cc:1035] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:27.023622 772 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.023795 367 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.024542 32761 raft_consensus.cc:2932] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.024750 32184 catalog_manager.cc:5484] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index 30 to 31, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New cstate: current_term: 2 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:27.025142 32168 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:27.025529 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 with delete_type TABLET_DATA_TOMBSTONED (TS e4df5ad2486247e89c2e44b31357188e not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:57340
I20240503 08:09:27.025830 777 tablet_replica.cc:330] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:27.025893 777 raft_consensus.cc:2229] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:27.025966 777 raft_consensus.cc:2258] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:27.026202 777 ts_tablet_manager.cc:1875] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:27.027508 777 ts_tablet_manager.cc:1888] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240503 08:09:27.027566 777 log.cc:1196] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P e4df5ad2486247e89c2e44b31357188e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/07d3a63eb2f640bc9f1ea96a6e071bd5
I20240503 08:09:27.027902 32169 catalog_manager.cc:4830] TS e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495): tablet 07d3a63eb2f640bc9f1ea96a6e071bd5 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:27.036520 766 raft_consensus.cc:979] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 2ac031e1b6da412da3f1c277da38d38b to start an election
I20240503 08:09:27.036641 766 raft_consensus.cc:1067] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Signalling peer 2ac031e1b6da412da3f1c277da38d38b to start an election
I20240503 08:09:27.036828 368 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508"
dest_uuid: "2ac031e1b6da412da3f1c277da38d38b"
from {username='slave'} at 127.0.0.1:53878
I20240503 08:09:27.036940 368 raft_consensus.cc:482] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:27.036983 368 raft_consensus.cc:3037] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:27.037972 368 raft_consensus.cc:504] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:27.038165 368 leader_election.cc:288] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [CANDIDATE]: Term 3 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:27.038305 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508" candidate_uuid: "2ac031e1b6da412da3f1c277da38d38b" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:27.038390 32434 raft_consensus.cc:3037] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:27.039253 32434 raft_consensus.cc:2454] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac031e1b6da412da3f1c277da38d38b in term 3.
I20240503 08:09:27.038305 32281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88518b63f713435f9d72e9cf2a395508" candidate_uuid: "2ac031e1b6da412da3f1c277da38d38b" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:27.039407 32281 raft_consensus.cc:3032] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:09:27.039414 332 leader_election.cc:302] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac031e1b6da412da3f1c277da38d38b, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:27.039428 32281 raft_consensus.cc:729] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:27.039530 32281 consensus_queue.cc:260] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:27.039551 751 raft_consensus.cc:2781] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:09:27.039633 751 raft_consensus.cc:686] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 3 LEADER]: Becoming Leader. State: Replica: 2ac031e1b6da412da3f1c277da38d38b, State: Running, Role: LEADER
I20240503 08:09:27.039631 32281 raft_consensus.cc:3037] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:27.039718 751 consensus_queue.cc:237] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:27.040269 32182 catalog_manager.cc:5484] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b reported cstate change: term changed from 2 to 3, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70). New cstate: current_term: 3 leader_uuid: "2ac031e1b6da412da3f1c277da38d38b" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:27.040467 32281 raft_consensus.cc:2454] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac031e1b6da412da3f1c277da38d38b in term 3.
I20240503 08:09:27.042156 573 raft_consensus.cc:1050] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: attempting to promote NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 to VOTER
I20240503 08:09:27.042320 573 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:27.042896 32434 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:27.042893 32761 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEARNER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:27.043015 32356 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:27.043107 781 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:27.043378 779 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:27.043501 573 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:27.044158 573 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.044260 32761 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.044425 32434 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.045329 32356 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.045382 32182 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:27.046244 32685 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:27.046697 32356 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:09:27.046728 32761 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:09:27.046906 779 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:27.047001 573 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:27.047595 573 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.047708 32761 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.047816 32356 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.048660 32227 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55c3a314149b4921b6cddce536a010e9 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:27.048748 32182 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 31 to 32, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New cstate: current_term: 2 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:27.049470 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 55c3a314149b4921b6cddce536a010e9 with delete_type TABLET_DATA_TOMBSTONED (TS e4df5ad2486247e89c2e44b31357188e not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:57340
I20240503 08:09:27.049593 777 tablet_replica.cc:330] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:27.049664 777 raft_consensus.cc:2229] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:27.049770 777 raft_consensus.cc:2258] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:27.050104 777 ts_tablet_manager.cc:1875] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:27.051571 777 ts_tablet_manager.cc:1888] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240503 08:09:27.051633 777 log.cc:1196] T 55c3a314149b4921b6cddce536a010e9 P e4df5ad2486247e89c2e44b31357188e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/55c3a314149b4921b6cddce536a010e9
I20240503 08:09:27.051966 32169 catalog_manager.cc:4830] TS e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495): tablet 55c3a314149b4921b6cddce536a010e9 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:27.094134 32360 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 7 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 32. Preceding OpId from leader: term: 7 index: 33. (index mismatch)
I20240503 08:09:27.094383 766 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:09:27.094732 32434 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 32. Preceding OpId from leader: term: 7 index: 33. (index mismatch)
I20240503 08:09:27.094929 766 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:09:27.095878 368 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:09:27.096786 368 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 7 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 6 index: 32. Preceding OpId from leader: term: 7 index: 33. (index mismatch)
I20240503 08:09:27.097129 715 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:09:27.144744 32434 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:09:27.145023 766 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:27.146281 32685 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 08:09:27.146572 726 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:27.318831 722 raft_consensus.cc:1050] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2: attempting to promote NON_VOTER 15ce703959504369884c5eac0c7fa77d to VOTER
I20240503 08:09:27.319068 722 consensus_queue.cc:237] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 26, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:27.319798 32281 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 08:09:27.319844 32685 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 3 LEARNER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 08:09:27.319919 32434 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 08:09:27.320031 770 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:27.320180 770 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:27.320482 772 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:27.321893 773 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 28 to 29, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:27.322038 32685 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:27.322965 32434 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:27.323256 32182 catalog_manager.cc:5484] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index 28 to 29, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:27.324244 32281 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:27.324363 32356 consensus_queue.cc:237] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:27.324712 32685 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:09:27.324826 32434 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:09:27.324946 770 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:27.325353 773 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:27.325742 770 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:27.325840 32685 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:27.326062 32434 raft_consensus.cc:2932] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:27.326826 32180 catalog_manager.cc:5484] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: config changed from index 29 to 30, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New cstate: current_term: 3 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:27.326871 32168 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4979660e62114184b95e7c2b04a26584 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:27.327602 32270 tablet_service.cc:1514] Processing DeleteTablet for tablet 4979660e62114184b95e7c2b04a26584 with delete_type TABLET_DATA_TOMBSTONED (TS 94175e832a7f4428be9adc9cfa9032d9 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:49090
I20240503 08:09:27.327908 794 tablet_replica.cc:330] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:27.328014 794 raft_consensus.cc:2229] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:27.328089 794 raft_consensus.cc:2258] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:27.328332 794 ts_tablet_manager.cc:1875] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:27.329780 794 ts_tablet_manager.cc:1888] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20240503 08:09:27.329840 794 log.cc:1196] T 4979660e62114184b95e7c2b04a26584 P 94175e832a7f4428be9adc9cfa9032d9: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/4979660e62114184b95e7c2b04a26584
I20240503 08:09:27.330188 32170 catalog_manager.cc:4830] TS 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181): tablet 4979660e62114184b95e7c2b04a26584 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:27.330480 32281 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 7.33, Last appended by leader: 32, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:27.330904 32434 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 7 index: 33. Preceding OpId from leader: term: 7 index: 34. (index mismatch)
I20240503 08:09:27.330922 367 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 7 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 7 index: 33. Preceding OpId from leader: term: 7 index: 34. (index mismatch)
I20240503 08:09:27.331338 766 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:09:27.331454 726 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:09:27.332224 715 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 7 LEADER]: Committing config change with OpId 7.34: config changed from index 32 to 34, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.332337 368 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 7 FOLLOWER]: Committing config change with OpId 7.34: config changed from index 32 to 34, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.332499 32434 raft_consensus.cc:2932] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Committing config change with OpId 7.34: config changed from index 32 to 34, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.333557 32170 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10e0365e2b664690ae51f079c4063b8e with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:09:27.333642 32182 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b reported cstate change: config changed from index 32 to 34, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New cstate: current_term: 7 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:27.334355 32352 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e0365e2b664690ae51f079c4063b8e with delete_type TABLET_DATA_TOMBSTONED (TS 441c21339c5c4941943e09a19ca62fe2 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:27.334465 769 tablet_replica.cc:330] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:27.334540 769 raft_consensus.cc:2229] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:27.334609 769 raft_consensus.cc:2258] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:27.334946 769 ts_tablet_manager.cc:1875] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:27.336030 769 ts_tablet_manager.cc:1888] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.33
I20240503 08:09:27.336081 769 log.cc:1196] T 10e0365e2b664690ae51f079c4063b8e P 441c21339c5c4941943e09a19ca62fe2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/10e0365e2b664690ae51f079c4063b8e
I20240503 08:09:27.336373 32168 catalog_manager.cc:4830] TS 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189): tablet 10e0365e2b664690ae51f079c4063b8e (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:27.345269 795 raft_consensus.cc:1050] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: attempting to promote NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 to VOTER
I20240503 08:09:27.345554 795 consensus_queue.cc:237] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:27.346062 32356 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:27.346062 32761 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEARNER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:27.346372 32281 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240503 08:09:27.346467 797 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:27.346864 795 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:27.347007 751 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:27.347765 751 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.348026 32356 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.349022 32184 catalog_manager.cc:5484] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b reported cstate change: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2ac031e1b6da412da3f1c277da38d38b" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:27.349180 32761 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.349401 32281 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.350874 368 consensus_queue.cc:237] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:27.351222 32761 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:09:27.351239 32356 raft_consensus.cc:1261] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240503 08:09:27.351444 795 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:27.351627 751 consensus_queue.cc:1035] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:27.352167 751 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.352277 32761 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.352990 32229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7ac923b6f4774b589c220467ccb7fd8f with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:27.353196 32356 raft_consensus.cc:2932] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.353257 32184 catalog_manager.cc:5484] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 reported cstate change: config changed from index 31 to 32, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New cstate: current_term: 2 leader_uuid: "2ac031e1b6da412da3f1c277da38d38b" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:27.354158 32270 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ac923b6f4774b589c220467ccb7fd8f with delete_type TABLET_DATA_TOMBSTONED (TS 94175e832a7f4428be9adc9cfa9032d9 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:49090
I20240503 08:09:27.354254 794 tablet_replica.cc:330] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:27.354326 794 raft_consensus.cc:2229] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:27.354446 794 raft_consensus.cc:2258] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:27.354688 794 ts_tablet_manager.cc:1875] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:27.355890 794 ts_tablet_manager.cc:1888] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240503 08:09:27.355940 794 log.cc:1196] T 7ac923b6f4774b589c220467ccb7fd8f P 94175e832a7f4428be9adc9cfa9032d9: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/7ac923b6f4774b589c220467ccb7fd8f
I20240503 08:09:27.356225 32170 catalog_manager.cc:4830] TS 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181): tablet 7ac923b6f4774b589c220467ccb7fd8f (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:27.534647 32281 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:09:27.534941 797 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:27.536139 32434 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:09:27.536314 720 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:28.354302 32685 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:28.355156 32356 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 08:09:28.355369 32761 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240503 08:09:28.355607 805 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:09:28.355741 804 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:09:28.356616 804 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:28.356767 32761 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:28.358105 32184 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 reported cstate change: config changed from index 32 to 33, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New cstate: current_term: 2 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:28.358161 32356 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } } }
I20240503 08:09:28.358569 367 consensus_queue.cc:237] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:28.359397 32281 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 08:09:28.359716 32434 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 08:09:28.359793 720 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:28.360289 720 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20240503 08:09:28.360868 32650 consensus_peers.cc:485] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d -> Peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543): Couldn't send request to peer 2ac031e1b6da412da3f1c277da38d38b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55c3a314149b4921b6cddce536a010e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:28.361127 751 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 28 to 31, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:28.361196 32434 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:28.362356 32283 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:28.362454 32182 catalog_manager.cc:5484] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index 28 to 31, NON_VOTER 15ce703959504369884c5eac0c7fa77d (127.28.55.68) added. New cstate: current_term: 3 leader_uuid: "2ac031e1b6da412da3f1c277da38d38b" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } } }
I20240503 08:09:28.362776 32281 consensus_queue.cc:237] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
W20240503 08:09:28.363596 331 consensus_peers.cc:485] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b -> Peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117): Couldn't send request to peer 15ce703959504369884c5eac0c7fa77d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88518b63f713435f9d72e9cf2a395508. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:28.363634 32434 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 08:09:28.363601 32685 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
W20240503 08:09:28.363842 32251 consensus_peers.cc:485] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 -> Peer 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733): Couldn't send request to peer 08561681ad9a4dc5be7c769e0a959ca9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eeef2123c7341509085d8ca8ff2b2ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:28.363842 766 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:28.364012 766 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:28.364745 766 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:28.365717 32434 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:28.365727 32685 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } } }
I20240503 08:09:28.365775 32180 catalog_manager.cc:5484] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: config changed from index 27 to 29, NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) added. New cstate: current_term: 2 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:28.680948 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:28.681844 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:28.682052 32283 raft_consensus.cc:595] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 7 LEADER]: Received request to transfer leadership to TS e4df5ad2486247e89c2e44b31357188e
I20240503 08:09:28.682365 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "15ce703959504369884c5eac0c7fa77d"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:28.682430 32283 raft_consensus.cc:595] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Received request to transfer leadership to TS 15ce703959504369884c5eac0c7fa77d
I20240503 08:09:28.682732 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 08:09:28.682834 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:28.805694 819 raft_consensus.cc:979] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:28.805799 819 raft_consensus.cc:1067] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 7 LEADER]: Signalling peer e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:28.805964 32434 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:57368
I20240503 08:09:28.806059 32434 raft_consensus.cc:482] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:28.806088 32434 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:09:28.806953 32434 raft_consensus.cc:504] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:28.807102 32434 leader_election.cc:288] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 8 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543)
I20240503 08:09:28.807250 367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 8 candidate_status { last_received { term: 7 index: 34 } } ignore_live_leader: true dest_uuid: "2ac031e1b6da412da3f1c277da38d38b"
I20240503 08:09:28.807274 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e0365e2b664690ae51f079c4063b8e" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 8 candidate_status { last_received { term: 7 index: 34 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:28.807315 367 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:09:28.807356 32283 raft_consensus.cc:3032] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 08:09:28.807374 32283 raft_consensus.cc:729] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:28.807413 32283 consensus_queue.cc:260] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 7.34, Last appended by leader: 34, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:28.807590 32283 raft_consensus.cc:3037] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:09:28.808051 367 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 8.
I20240503 08:09:28.808143 32283 raft_consensus.cc:2454] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 8.
I20240503 08:09:28.808271 32402 leader_election.cc:302] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac031e1b6da412da3f1c277da38d38b, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:28.808367 753 raft_consensus.cc:2781] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 8 FOLLOWER]: Leader election won for term 8
I20240503 08:09:28.808534 753 raft_consensus.cc:686] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 8 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:28.808598 753 consensus_queue.cc:237] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 7.34, Last appended by leader: 34, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:28.809098 32184 catalog_manager.cc:5484] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 7 to 8, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 8 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:28.848918 818 raft_consensus.cc:979] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 15ce703959504369884c5eac0c7fa77d to start an election
I20240503 08:09:28.849012 818 raft_consensus.cc:1067] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Signalling peer 15ce703959504369884c5eac0c7fa77d to start an election
I20240503 08:09:28.849169 32685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad"
dest_uuid: "15ce703959504369884c5eac0c7fa77d"
from {username='slave'} at 127.0.0.1:57958
I20240503 08:09:28.849272 32685 raft_consensus.cc:482] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:28.849328 32685 raft_consensus.cc:3037] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:28.850162 32685 raft_consensus.cc:504] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:28.850353 32685 leader_election.cc:288] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [CANDIDATE]: Term 3 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:28.850504 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad" candidate_uuid: "15ce703959504369884c5eac0c7fa77d" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:28.850581 32434 raft_consensus.cc:3037] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:28.850551 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eeef2123c7341509085d8ca8ff2b2ad" candidate_uuid: "15ce703959504369884c5eac0c7fa77d" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:28.850611 32283 raft_consensus.cc:3032] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:09:28.850642 32283 raft_consensus.cc:729] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:28.850703 32283 consensus_queue.cc:260] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:28.850826 32283 raft_consensus.cc:3037] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:28.851295 32434 raft_consensus.cc:2454] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ce703959504369884c5eac0c7fa77d in term 3.
I20240503 08:09:28.851361 32283 raft_consensus.cc:2454] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ce703959504369884c5eac0c7fa77d in term 3.
I20240503 08:09:28.851476 32650 leader_election.cc:302] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ce703959504369884c5eac0c7fa77d, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:28.851569 804 raft_consensus.cc:2781] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:09:28.851621 804 raft_consensus.cc:686] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 3 LEADER]: Becoming Leader. State: Replica: 15ce703959504369884c5eac0c7fa77d, State: Running, Role: LEADER
I20240503 08:09:28.851671 804 consensus_queue.cc:237] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:28.852169 32184 catalog_manager.cc:5484] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d reported cstate change: term changed from 2 to 3, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 15ce703959504369884c5eac0c7fa77d (127.28.55.68). New cstate: current_term: 3 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:28.897742 824 ts_tablet_manager.cc:902] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: Initiating tablet copy from peer 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543)
I20240503 08:09:28.898133 824 tablet_copy_client.cc:308] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.70:33543
I20240503 08:09:28.898414 378 tablet_copy_service.cc:140] P 2ac031e1b6da412da3f1c277da38d38b: Received BeginTabletCopySession request for tablet 88518b63f713435f9d72e9cf2a395508 from peer 15ce703959504369884c5eac0c7fa77d ({username='slave'} at 127.0.0.1:45462)
I20240503 08:09:28.898470 378 tablet_copy_service.cc:161] P 2ac031e1b6da412da3f1c277da38d38b: Beginning new tablet copy session on tablet 88518b63f713435f9d72e9cf2a395508 from peer 15ce703959504369884c5eac0c7fa77d at {username='slave'} at 127.0.0.1:45462: session id = 15ce703959504369884c5eac0c7fa77d-88518b63f713435f9d72e9cf2a395508
I20240503 08:09:28.899016 378 tablet_copy_source_session.cc:205] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:28.899396 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88518b63f713435f9d72e9cf2a395508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:28.900614 824 tablet_copy_client.cc:783] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:28.900774 824 tablet_copy_client.cc:647] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:28.901353 828 tablet_copy_client.cc:1002] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:28.952473 829 ts_tablet_manager.cc:902] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: Initiating tablet copy from peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117)
I20240503 08:09:28.952850 829 tablet_copy_client.cc:308] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.68:43117
I20240503 08:09:28.953052 32696 tablet_copy_service.cc:140] P 15ce703959504369884c5eac0c7fa77d: Received BeginTabletCopySession request for tablet 55c3a314149b4921b6cddce536a010e9 from peer 2ac031e1b6da412da3f1c277da38d38b ({username='slave'} at 127.0.0.1:41470)
I20240503 08:09:28.953105 32696 tablet_copy_service.cc:161] P 15ce703959504369884c5eac0c7fa77d: Beginning new tablet copy session on tablet 55c3a314149b4921b6cddce536a010e9 from peer 2ac031e1b6da412da3f1c277da38d38b at {username='slave'} at 127.0.0.1:41470: session id = 2ac031e1b6da412da3f1c277da38d38b-55c3a314149b4921b6cddce536a010e9
I20240503 08:09:28.953734 32696 tablet_copy_source_session.cc:205] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:28.954066 829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c3a314149b4921b6cddce536a010e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:28.955277 829 tablet_copy_client.cc:783] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:28.955391 829 tablet_copy_client.cc:647] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:28.955910 833 tablet_copy_client.cc:1002] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:29.213682 367 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 8 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 7 index: 34. Preceding OpId from leader: term: 8 index: 35. (index mismatch)
I20240503 08:09:29.214037 753 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:09:29.215394 32283 raft_consensus.cc:1261] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 8 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 7 index: 34. Preceding OpId from leader: term: 8 index: 35. (index mismatch)
I20240503 08:09:29.215591 836 consensus_queue.cc:1035] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:09:29.272047 32283 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 08:09:29.272267 805 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:29.273615 32434 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20240503 08:09:29.273635 32650 consensus_peers.cc:485] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d -> Peer 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69:45733): Couldn't send request to peer 08561681ad9a4dc5be7c769e0a959ca9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eeef2123c7341509085d8ca8ff2b2ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:29.273962 805 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:29.677863 843 ts_tablet_manager.cc:902] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: Initiating tablet copy from peer 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117)
I20240503 08:09:29.678300 843 tablet_copy_client.cc:308] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.68:43117
I20240503 08:09:29.678493 32696 tablet_copy_service.cc:140] P 15ce703959504369884c5eac0c7fa77d: Received BeginTabletCopySession request for tablet 4eeef2123c7341509085d8ca8ff2b2ad from peer 08561681ad9a4dc5be7c769e0a959ca9 ({username='slave'} at 127.0.0.1:57978)
I20240503 08:09:29.678546 32696 tablet_copy_service.cc:161] P 15ce703959504369884c5eac0c7fa77d: Beginning new tablet copy session on tablet 4eeef2123c7341509085d8ca8ff2b2ad from peer 08561681ad9a4dc5be7c769e0a959ca9 at {username='slave'} at 127.0.0.1:57978: session id = 08561681ad9a4dc5be7c769e0a959ca9-4eeef2123c7341509085d8ca8ff2b2ad
I20240503 08:09:29.679279 32696 tablet_copy_source_session.cc:205] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:29.679612 843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eeef2123c7341509085d8ca8ff2b2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:29.681289 843 tablet_copy_client.cc:783] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:29.681418 843 tablet_copy_client.cc:647] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:29.681916 847 tablet_copy_client.cc:1002] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Injecting latency into file download: 2000
I20240503 08:09:30.683041 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:30.684242 32434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:30.684355 32434 raft_consensus.cc:595] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 3 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:30.684518 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:30.684572 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:30.738205 822 raft_consensus.cc:979] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:30.738301 822 raft_consensus.cc:1067] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 3 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:30.738490 32283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49104
I20240503 08:09:30.738580 32283 raft_consensus.cc:482] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:30.738603 32283 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:30.739694 32283 raft_consensus.cc:504] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:30.739892 32283 leader_election.cc:288] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 4 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:30.740001 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:30.740064 32434 raft_consensus.cc:3032] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:09:30.740010 32356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:30.740092 32434 raft_consensus.cc:729] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:30.740106 32356 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:30.740141 32434 consensus_queue.cc:260] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:30.740242 32434 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:30.740902 32356 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 4.
I20240503 08:09:30.740984 32434 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 4.
I20240503 08:09:30.741032 32250 leader_election.cc:302] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:30.741103 818 raft_consensus.cc:2781] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:09:30.741209 818 raft_consensus.cc:686] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:30.741262 818 consensus_queue.cc:237] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 27, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:30.741736 32180 catalog_manager.cc:5484] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 3 to 4, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 4 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:30.901628 824 tablet_copy_client.cc:523] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:30.902720 824 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: Bootstrap starting.
I20240503 08:09:30.907122 824 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:30.907397 824 tablet_bootstrap.cc:492] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: Bootstrap complete.
I20240503 08:09:30.907533 824 ts_tablet_manager.cc:1367] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:09:30.907727 824 raft_consensus.cc:348] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:30.907824 824 raft_consensus.cc:729] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 15ce703959504369884c5eac0c7fa77d, State: Initialized, Role: LEARNER
I20240503 08:09:30.907866 824 consensus_queue.cc:260] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: true } }
I20240503 08:09:30.907977 824 ts_tablet_manager.cc:1398] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:30.908190 378 tablet_copy_service.cc:342] P 2ac031e1b6da412da3f1c277da38d38b: Request end of tablet copy session 15ce703959504369884c5eac0c7fa77d-88518b63f713435f9d72e9cf2a395508 received from {username='slave'} at 127.0.0.1:45462
I20240503 08:09:30.908267 378 tablet_copy_service.cc:433] P 2ac031e1b6da412da3f1c277da38d38b: ending tablet copy session 15ce703959504369884c5eac0c7fa77d-88518b63f713435f9d72e9cf2a395508 on tablet 88518b63f713435f9d72e9cf2a395508 with peer 15ce703959504369884c5eac0c7fa77d
I20240503 08:09:30.956156 829 tablet_copy_client.cc:523] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:30.957191 829 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap starting.
I20240503 08:09:30.961256 829 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=89 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:30.961499 829 tablet_bootstrap.cc:492] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: Bootstrap complete.
I20240503 08:09:30.961616 829 ts_tablet_manager.cc:1367] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:09:30.961803 829 raft_consensus.cc:348] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:30.961906 829 raft_consensus.cc:729] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2ac031e1b6da412da3f1c277da38d38b, State: Initialized, Role: LEARNER
I20240503 08:09:30.961942 829 consensus_queue.cc:260] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: NON_VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: true } }
I20240503 08:09:30.962069 829 ts_tablet_manager.cc:1398] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: Time spent starting tablet: real 0.000s user 0.000s sys 0.003s
I20240503 08:09:30.962227 32696 tablet_copy_service.cc:342] P 15ce703959504369884c5eac0c7fa77d: Request end of tablet copy session 2ac031e1b6da412da3f1c277da38d38b-55c3a314149b4921b6cddce536a010e9 received from {username='slave'} at 127.0.0.1:41470
I20240503 08:09:30.962292 32696 tablet_copy_service.cc:433] P 15ce703959504369884c5eac0c7fa77d: ending tablet copy session 2ac031e1b6da412da3f1c277da38d38b-55c3a314149b4921b6cddce536a010e9 on tablet 55c3a314149b4921b6cddce536a010e9 with peer 2ac031e1b6da412da3f1c277da38d38b
I20240503 08:09:31.233561 368 raft_consensus.cc:1203] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.32->[2.33-2.33] Dedup: 2.33->[]
I20240503 08:09:31.262137 32356 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240503 08:09:31.262387 818 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:31.263478 32434 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 4 index: 28. (index mismatch)
I20240503 08:09:31.263659 860 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:31.286437 32686 raft_consensus.cc:1203] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [term 3 LEARNER]: Deduplicated request from leader. Original: 3.30->[3.31-3.31] Dedup: 3.31->[]
I20240503 08:09:31.682116 843 tablet_copy_client.cc:523] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:31.683020 843 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap starting.
I20240503 08:09:31.686319 843 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=92 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:31.686511 843 tablet_bootstrap.cc:492] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: Bootstrap complete.
I20240503 08:09:31.686621 843 ts_tablet_manager.cc:1367] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:09:31.686784 843 raft_consensus.cc:348] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:31.686894 843 raft_consensus.cc:729] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 08561681ad9a4dc5be7c769e0a959ca9, State: Initialized, Role: LEARNER
I20240503 08:09:31.686926 843 consensus_queue.cc:260] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: NON_VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: true } }
I20240503 08:09:31.687050 843 ts_tablet_manager.cc:1398] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: Time spent starting tablet: real 0.000s user 0.001s sys 0.002s
I20240503 08:09:31.687285 32696 tablet_copy_service.cc:342] P 15ce703959504369884c5eac0c7fa77d: Request end of tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-4eeef2123c7341509085d8ca8ff2b2ad received from {username='slave'} at 127.0.0.1:57978
I20240503 08:09:31.687340 32696 tablet_copy_service.cc:433] P 15ce703959504369884c5eac0c7fa77d: ending tablet copy session 08561681ad9a4dc5be7c769e0a959ca9-4eeef2123c7341509085d8ca8ff2b2ad on tablet 4eeef2123c7341509085d8ca8ff2b2ad with peer 08561681ad9a4dc5be7c769e0a959ca9
I20240503 08:09:31.721135 868 raft_consensus.cc:1050] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: attempting to promote NON_VOTER 15ce703959504369884c5eac0c7fa77d to VOTER
I20240503 08:09:31.721406 868 consensus_queue.cc:237] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:31.721937 32685 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [term 3 LEARNER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 08:09:31.721937 32434 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 08:09:31.722007 32283 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 08:09:31.722226 849 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:31.722349 868 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:31.722469 867 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:31.723533 867 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:31.723680 32434 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:31.723680 32686 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:31.724879 32180 catalog_manager.cc:5484] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e reported cstate change: config changed from index 31 to 32, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "2ac031e1b6da412da3f1c277da38d38b" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:31.725134 32283 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 15ce703959504369884c5eac0c7fa77d (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } attrs { replace: true } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
W20240503 08:09:31.726181 868 consensus_queue.cc:1159] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "94175e832a7f4428be9adc9cfa9032d9" responder_term: 3 status { last_received { term: 3 index: 32 } last_committed_idx: 32 last_received_current_leader { term: 3 index: 32 } }
I20240503 08:09:31.726301 367 consensus_queue.cc:237] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:31.726675 32686 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 08:09:31.726718 32434 raft_consensus.cc:1261] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 2ac031e1b6da412da3f1c277da38d38b: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240503 08:09:31.726955 868 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:09:31.727062 849 consensus_queue.cc:1035] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 08:09:31.727684 870 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:31.727823 32685 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:31.728900 32229 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88518b63f713435f9d72e9cf2a395508 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:31.729111 32184 catalog_manager.cc:5484] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 32 to 33, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New cstate: current_term: 3 leader_uuid: "2ac031e1b6da412da3f1c277da38d38b" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:31.729686 32434 raft_consensus.cc:2932] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } }
I20240503 08:09:31.730664 32270 tablet_service.cc:1514] Processing DeleteTablet for tablet 88518b63f713435f9d72e9cf2a395508 with delete_type TABLET_DATA_TOMBSTONED (TS 94175e832a7f4428be9adc9cfa9032d9 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:49090
I20240503 08:09:31.730967 877 tablet_replica.cc:330] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:31.731132 877 raft_consensus.cc:2229] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:31.731217 877 raft_consensus.cc:2258] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:31.731550 877 ts_tablet_manager.cc:1875] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:31.733366 877 ts_tablet_manager.cc:1888] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20240503 08:09:31.733435 877 log.cc:1196] T 88518b63f713435f9d72e9cf2a395508 P 94175e832a7f4428be9adc9cfa9032d9: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/88518b63f713435f9d72e9cf2a395508
I20240503 08:09:31.733896 32170 catalog_manager.cc:4830] TS 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181): tablet 88518b63f713435f9d72e9cf2a395508 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:31.743131 878 raft_consensus.cc:1050] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: attempting to promote NON_VOTER 2ac031e1b6da412da3f1c277da38d38b to VOTER
I20240503 08:09:31.743468 878 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:31.744102 32356 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 08:09:31.744140 32761 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 08:09:31.744254 367 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEARNER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240503 08:09:31.744310 881 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:09:31.744489 878 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:09:31.744611 841 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 08:09:31.745568 881 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:31.745700 367 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:31.745787 32761 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:31.746769 32356 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:31.746841 32184 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 33 to 34, 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:31.748118 32685 consensus_queue.cc:237] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:31.748484 367 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240503 08:09:31.748495 32761 raft_consensus.cc:1261] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240503 08:09:31.748675 880 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:09:31.748836 878 consensus_queue.cc:1035] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 08:09:31.749336 878 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:31.749562 367 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:31.749565 32761 raft_consensus.cc:2932] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:31.750447 32227 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55c3a314149b4921b6cddce536a010e9 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:31.750540 32182 catalog_manager.cc:5484] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b reported cstate change: config changed from index 34 to 35, VOTER 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) evicted. New cstate: current_term: 2 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } } }
I20240503 08:09:31.751345 32352 tablet_service.cc:1514] Processing DeleteTablet for tablet 55c3a314149b4921b6cddce536a010e9 with delete_type TABLET_DATA_TOMBSTONED (TS 441c21339c5c4941943e09a19ca62fe2 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:50228
I20240503 08:09:31.751585 885 tablet_replica.cc:330] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:31.751657 885 raft_consensus.cc:2229] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:31.751770 885 raft_consensus.cc:2258] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:31.752066 885 ts_tablet_manager.cc:1875] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:31.753548 885 ts_tablet_manager.cc:1888] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.34
I20240503 08:09:31.753609 885 log.cc:1196] T 55c3a314149b4921b6cddce536a010e9 P 441c21339c5c4941943e09a19ca62fe2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/55c3a314149b4921b6cddce536a010e9
I20240503 08:09:31.753962 32168 catalog_manager.cc:4830] TS 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189): tablet 55c3a314149b4921b6cddce536a010e9 (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:32.050204 32761 raft_consensus.cc:1203] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEARNER]: Deduplicated request from leader. Original: 2.29->[3.30-3.30] Dedup: 3.30->[]
I20240503 08:09:32.547415 880 raft_consensus.cc:1050] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: attempting to promote NON_VOTER 08561681ad9a4dc5be7c769e0a959ca9 to VOTER
I20240503 08:09:32.547677 880 consensus_queue.cc:237] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:32.548238 32761 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEARNER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 08:09:32.548427 32283 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 08:09:32.548446 888 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:32.548548 32434 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 08:09:32.548710 888 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:32.548941 891 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:32.550303 880 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:32.550460 32761 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:32.550981 32434 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:32.551723 32184 catalog_manager.cc:5484] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 29 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:32.552227 32283 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } attrs { replace: true } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:32.553020 32685 consensus_queue.cc:237] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } }
I20240503 08:09:32.553499 32761 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 08:09:32.553511 32283 raft_consensus.cc:1261] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer 15ce703959504369884c5eac0c7fa77d: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 08:09:32.553674 889 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:32.553808 891 consensus_queue.cc:1035] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:32.554343 880 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:32.554472 32283 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:32.554996 32761 raft_consensus.cc:2932] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } }
I20240503 08:09:32.555286 32227 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4eeef2123c7341509085d8ca8ff2b2ad with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:32.556073 32180 catalog_manager.cc:5484] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d reported cstate change: config changed from index 31 to 32, VOTER e4df5ad2486247e89c2e44b31357188e (127.28.55.67) evicted. New cstate: current_term: 3 leader_uuid: "15ce703959504369884c5eac0c7fa77d" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:09:32.556830 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eeef2123c7341509085d8ca8ff2b2ad with delete_type TABLET_DATA_TOMBSTONED (TS e4df5ad2486247e89c2e44b31357188e not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:57340
I20240503 08:09:32.557049 899 tablet_replica.cc:330] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:32.557117 899 raft_consensus.cc:2229] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:32.557189 899 raft_consensus.cc:2258] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:32.557417 899 ts_tablet_manager.cc:1875] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:32.558604 899 ts_tablet_manager.cc:1888] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20240503 08:09:32.558657 899 log.cc:1196] T 4eeef2123c7341509085d8ca8ff2b2ad P e4df5ad2486247e89c2e44b31357188e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/4eeef2123c7341509085d8ca8ff2b2ad
I20240503 08:09:32.558997 32169 catalog_manager.cc:4830] TS e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495): tablet 4eeef2123c7341509085d8ca8ff2b2ad (table test-workload [id=79e6b61eb755449b82517db4a1ad4b89]) successfully deleted
I20240503 08:09:32.684954 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:32.686256 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:32.686357 32283 raft_consensus.cc:595] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:32.686542 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:32.686599 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:33.176193 887 raft_consensus.cc:979] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:33.176290 887 raft_consensus.cc:1067] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:33.176504 32356 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50240
I20240503 08:09:33.176573 32356 raft_consensus.cc:482] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:33.176592 32356 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:33.177277 32356 raft_consensus.cc:504] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:33.177439 32356 leader_election.cc:288] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 5 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:33.177601 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:33.177660 32283 raft_consensus.cc:3032] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:09:33.177632 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:33.177716 32283 raft_consensus.cc:729] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:33.177734 32434 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:33.177776 32283 consensus_queue.cc:260] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:33.177858 32283 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:33.178344 32434 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 5.
I20240503 08:09:33.178368 32283 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 5.
I20240503 08:09:33.178560 32328 leader_election.cc:302] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:33.178709 773 raft_consensus.cc:2781] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:09:33.178830 773 raft_consensus.cc:686] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:33.178900 773 consensus_queue.cc:237] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 4.28, Last appended by leader: 28, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:33.179440 32182 catalog_manager.cc:5484] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 4 to 5, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 5 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:33.679291 32283 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240503 08:09:33.679519 773 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:33.680881 32434 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 4 index: 28. Preceding OpId from leader: term: 5 index: 29. (index mismatch)
I20240503 08:09:33.681088 909 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 08:09:34.686995 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:34.687994 32356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:34.688088 32356 raft_consensus.cc:595] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:34.688285 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:34.688349 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:34.688513 907 raft_consensus.cc:979] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:34.688575 907 raft_consensus.cc:1067] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:34.688741 32283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49120
I20240503 08:09:34.688795 32283 raft_consensus.cc:482] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:34.688823 32283 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:09:34.689513 32283 raft_consensus.cc:504] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:34.689666 32283 leader_election.cc:288] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 6 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:34.689860 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:34.689875 32356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:34.689950 32434 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:09:34.689957 32356 raft_consensus.cc:3032] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:09:34.690073 32356 raft_consensus.cc:729] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:34.690146 32356 consensus_queue.cc:260] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:34.690232 32356 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:09:34.690663 32434 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 6.
I20240503 08:09:34.690758 32356 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 6.
I20240503 08:09:34.690884 32251 leader_election.cc:302] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:34.691023 887 raft_consensus.cc:2781] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:09:34.691124 887 raft_consensus.cc:686] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:34.691184 887 consensus_queue.cc:237] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 5.29, Last appended by leader: 29, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:34.691712 32182 catalog_manager.cc:5484] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 5 to 6, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 6 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:35.093359 32356 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20240503 08:09:35.093755 887 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:35.095060 32434 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 6 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 5 index: 29. Preceding OpId from leader: term: 6 index: 30. (index mismatch)
I20240503 08:09:35.095284 887 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 08:09:36.688740 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:36.689383 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:36.689469 32283 raft_consensus.cc:595] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:36.689661 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:36.689709 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:37.055429 925 raft_consensus.cc:979] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:37.055518 925 raft_consensus.cc:1067] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:37.055708 32356 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50240
I20240503 08:09:37.055775 32356 raft_consensus.cc:482] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:37.055794 32356 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:09:37.056476 32356 raft_consensus.cc:504] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:37.056600 32356 leader_election.cc:288] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 7 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495)
I20240503 08:09:37.056771 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:37.056856 32283 raft_consensus.cc:3032] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:09:37.056866 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0b30e22bc04f638c852c02354109b6" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 7 candidate_status { last_received { term: 6 index: 30 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:37.056887 32283 raft_consensus.cc:729] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:37.056933 32434 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:09:37.056967 32283 consensus_queue.cc:260] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 6.30, Last appended by leader: 30, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:37.057061 32283 raft_consensus.cc:3037] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:09:37.057739 32434 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 7.
I20240503 08:09:37.057739 32283 raft_consensus.cc:2454] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 7.
I20240503 08:09:37.057951 32327 leader_election.cc:302] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:37.058099 907 raft_consensus.cc:2781] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:09:37.058216 907 raft_consensus.cc:686] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 7 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:37.058279 907 consensus_queue.cc:237] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 6.30, Last appended by leader: 30, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } }
I20240503 08:09:37.058899 32180 catalog_manager.cc:5484] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 6 to 7, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 7 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:37.514673 32283 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 7 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 7 index: 31. (index mismatch)
I20240503 08:09:37.514957 907 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:37.516105 32434 raft_consensus.cc:1261] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 6 index: 30. Preceding OpId from leader: term: 7 index: 31. (index mismatch)
I20240503 08:09:37.516317 927 consensus_queue.cc:1035] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:38.689961 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:38.690532 32356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4979660e62114184b95e7c2b04a26584"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:38.690616 32356 raft_consensus.cc:595] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Received request to transfer leadership to TS e4df5ad2486247e89c2e44b31357188e
I20240503 08:09:38.690862 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:38.690915 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:39.232410 938 raft_consensus.cc:979] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:39.232502 938 raft_consensus.cc:1067] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Signalling peer e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:39.232707 32434 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4979660e62114184b95e7c2b04a26584"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:57356
I20240503 08:09:39.232800 32434 raft_consensus.cc:482] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:39.232821 32434 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:39.233538 32434 raft_consensus.cc:504] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:39.233707 32434 leader_election.cc:288] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 4 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117)
I20240503 08:09:39.233933 32686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "15ce703959504369884c5eac0c7fa77d"
I20240503 08:09:39.233933 32356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:39.234046 32686 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:39.234051 32356 raft_consensus.cc:3032] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:09:39.234122 32356 raft_consensus.cc:729] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:39.234191 32356 consensus_queue.cc:260] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:39.234318 32356 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:09:39.234926 32686 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 4.
I20240503 08:09:39.235031 32356 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 4.
I20240503 08:09:39.235163 32401 leader_election.cc:302] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ce703959504369884c5eac0c7fa77d, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:39.235270 822 raft_consensus.cc:2781] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:09:39.235430 822 raft_consensus.cc:686] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 4 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:39.235513 822 consensus_queue.cc:237] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:39.236047 32182 catalog_manager.cc:5484] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 3 to 4, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 4 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:39.665819 32686 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 4 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:09:39.666080 822 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:39.667301 32356 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 08:09:39.667497 822 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 08:09:40.691084 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:40.691634 32434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:40.691728 32434 raft_consensus.cc:595] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 2 LEADER]: Received request to transfer leadership to TS 94175e832a7f4428be9adc9cfa9032d9
I20240503 08:09:40.691919 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:40.691983 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:41.057672 951 raft_consensus.cc:979] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:41.057762 951 raft_consensus.cc:1067] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 2 LEADER]: Signalling peer 94175e832a7f4428be9adc9cfa9032d9 to start an election
I20240503 08:09:41.058019 32283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
from {username='slave'} at 127.0.0.1:49104
I20240503 08:09:41.058125 32283 raft_consensus.cc:482] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:41.058147 32283 raft_consensus.cc:3037] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:41.058967 32283 raft_consensus.cc:504] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:41.059134 32283 leader_election.cc:288] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 3 election: Requested vote from peers e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495), 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189)
I20240503 08:09:41.059283 32356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:41.059278 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22779123b80e4ea4a396192e72b73fe8" candidate_uuid: "94175e832a7f4428be9adc9cfa9032d9" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:41.059396 32356 raft_consensus.cc:3037] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:41.059419 32434 raft_consensus.cc:3032] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:09:41.059450 32434 raft_consensus.cc:729] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:41.059504 32434 consensus_queue.cc:260] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:41.059581 32434 raft_consensus.cc:3037] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:41.060127 32356 raft_consensus.cc:2454] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 3.
I20240503 08:09:41.060137 32434 raft_consensus.cc:2454] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94175e832a7f4428be9adc9cfa9032d9 in term 3.
I20240503 08:09:41.060325 32250 leader_election.cc:302] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441c21339c5c4941943e09a19ca62fe2, 94175e832a7f4428be9adc9cfa9032d9; no voters:
I20240503 08:09:41.060418 925 raft_consensus.cc:2781] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:09:41.060514 925 raft_consensus.cc:686] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 LEADER]: Becoming Leader. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:41.060570 925 consensus_queue.cc:237] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } }
I20240503 08:09:41.061118 32182 catalog_manager.cc:5484] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 reported cstate change: term changed from 2 to 3, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65). New cstate: current_term: 3 leader_uuid: "94175e832a7f4428be9adc9cfa9032d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:41.668684 32356 raft_consensus.cc:1261] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 08:09:41.668931 925 consensus_queue.cc:1035] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:41.670253 32434 raft_consensus.cc:1261] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Refusing update from remote peer 94175e832a7f4428be9adc9cfa9032d9: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20240503 08:09:41.670527 925 consensus_queue.cc:1035] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 08:09:42.692234 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:42.693290 32283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263"
dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
mode: GRACEFUL
new_leader_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:57508
I20240503 08:09:42.693410 32283 raft_consensus.cc:595] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Received request to transfer leadership to TS e4df5ad2486247e89c2e44b31357188e
I20240503 08:09:42.693675 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:42.693745 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:42.898114 965 raft_consensus.cc:979] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9: : Instructing follower e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:42.898226 965 raft_consensus.cc:1067] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Signalling peer e4df5ad2486247e89c2e44b31357188e to start an election
I20240503 08:09:42.898452 32434 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
from {username='slave'} at 127.0.0.1:57368
I20240503 08:09:42.898545 32434 raft_consensus.cc:482] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:42.898564 32434 raft_consensus.cc:3037] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:42.899341 32434 raft_consensus.cc:504] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:42.899502 32434 leader_election.cc:288] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 3 election: Requested vote from peers 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181), 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117)
I20240503 08:09:42.899669 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "94175e832a7f4428be9adc9cfa9032d9"
I20240503 08:09:42.899753 32283 raft_consensus.cc:3032] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:09:42.899667 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c72a8ae02442ca1966c3536976263" candidate_uuid: "e4df5ad2486247e89c2e44b31357188e" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "15ce703959504369884c5eac0c7fa77d"
I20240503 08:09:42.899776 32283 raft_consensus.cc:729] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94175e832a7f4428be9adc9cfa9032d9, State: Running, Role: LEADER
I20240503 08:09:42.899807 32685 raft_consensus.cc:3037] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:42.899852 32283 consensus_queue.cc:260] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:42.899946 32283 raft_consensus.cc:3037] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:42.900478 32283 raft_consensus.cc:2454] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 3.
I20240503 08:09:42.900485 32685 raft_consensus.cc:2454] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4df5ad2486247e89c2e44b31357188e in term 3.
I20240503 08:09:42.900674 32403 leader_election.cc:302] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94175e832a7f4428be9adc9cfa9032d9, e4df5ad2486247e89c2e44b31357188e; no voters:
I20240503 08:09:42.900764 951 raft_consensus.cc:2781] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:09:42.900884 951 raft_consensus.cc:686] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 3 LEADER]: Becoming Leader. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:42.900944 951 consensus_queue.cc:237] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:42.901517 32182 catalog_manager.cc:5484] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e reported cstate change: term changed from 2 to 3, leader changed from 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65) to e4df5ad2486247e89c2e44b31357188e (127.28.55.67). New cstate: current_term: 3 leader_uuid: "e4df5ad2486247e89c2e44b31357188e" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:43.373638 32283 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:09:43.373950 951 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94175e832a7f4428be9adc9cfa9032d9" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:43.375370 32685 raft_consensus.cc:1261] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Refusing update from remote peer e4df5ad2486247e89c2e44b31357188e: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 08:09:43.375612 976 consensus_queue.cc:1035] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 08:09:44.694007 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:44.694581 32434 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4979660e62114184b95e7c2b04a26584"
dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
mode: GRACEFUL
new_leader_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:57390
I20240503 08:09:44.694680 32434 raft_consensus.cc:595] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 4 LEADER]: Received request to transfer leadership to TS 441c21339c5c4941943e09a19ca62fe2
I20240503 08:09:44.694881 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:44.694954 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:45.106274 967 raft_consensus.cc:979] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e: : Instructing follower 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:45.106369 967 raft_consensus.cc:1067] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 4 LEADER]: Signalling peer 441c21339c5c4941943e09a19ca62fe2 to start an election
I20240503 08:09:45.106555 32356 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4979660e62114184b95e7c2b04a26584"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
from {username='slave'} at 127.0.0.1:50236
I20240503 08:09:45.106647 32356 raft_consensus.cc:482] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:45.106667 32356 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:45.107350 32356 raft_consensus.cc:504] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:45.107506 32356 leader_election.cc:288] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 5 election: Requested vote from peers e4df5ad2486247e89c2e44b31357188e (127.28.55.67:44495), 15ce703959504369884c5eac0c7fa77d (127.28.55.68:43117)
I20240503 08:09:45.107658 32434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "e4df5ad2486247e89c2e44b31357188e"
I20240503 08:09:45.107690 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979660e62114184b95e7c2b04a26584" candidate_uuid: "441c21339c5c4941943e09a19ca62fe2" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "15ce703959504369884c5eac0c7fa77d"
I20240503 08:09:45.107739 32434 raft_consensus.cc:3032] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:09:45.107744 32685 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:45.107774 32434 raft_consensus.cc:729] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e4df5ad2486247e89c2e44b31357188e, State: Running, Role: LEADER
I20240503 08:09:45.107829 32434 consensus_queue.cc:260] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:45.107914 32434 raft_consensus.cc:3037] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:09:45.108489 32685 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 5.
I20240503 08:09:45.108645 32434 raft_consensus.cc:2454] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441c21339c5c4941943e09a19ca62fe2 in term 5.
I20240503 08:09:45.108673 32326 leader_election.cc:302] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ce703959504369884c5eac0c7fa77d, 441c21339c5c4941943e09a19ca62fe2; no voters:
I20240503 08:09:45.108824 938 raft_consensus.cc:2781] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:09:45.108942 938 raft_consensus.cc:686] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Becoming Leader. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:45.109004 938 consensus_queue.cc:237] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } }
I20240503 08:09:45.109568 32180 catalog_manager.cc:5484] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 reported cstate change: term changed from 4 to 5, leader changed from e4df5ad2486247e89c2e44b31357188e (127.28.55.67) to 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66). New cstate: current_term: 5 leader_uuid: "441c21339c5c4941943e09a19ca62fe2" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:45.618459 32434 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 08:09:45.618772 980 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4df5ad2486247e89c2e44b31357188e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:45.620016 32685 raft_consensus.cc:1261] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 5 FOLLOWER]: Refusing update from remote peer 441c21339c5c4941943e09a19ca62fe2: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20240503 08:09:45.620219 990 consensus_queue.cc:1035] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ce703959504369884c5eac0c7fa77d" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 08:09:46.695216 32238 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:46.696681 32356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5"
dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
mode: GRACEFUL
new_leader_uuid: "08561681ad9a4dc5be7c769e0a959ca9"
from {username='slave'} at 127.0.0.1:49498
I20240503 08:09:46.696851 32356 raft_consensus.cc:595] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Received request to transfer leadership to TS 08561681ad9a4dc5be7c769e0a959ca9
I20240503 08:09:46.697081 32238 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:46.697146 32238 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:46.833135 987 raft_consensus.cc:979] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2: : Instructing follower 08561681ad9a4dc5be7c769e0a959ca9 to start an election
I20240503 08:09:46.833247 987 raft_consensus.cc:1067] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Signalling peer 08561681ad9a4dc5be7c769e0a959ca9 to start an election
I20240503 08:09:46.833488 32761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5"
dest_uuid: "08561681ad9a4dc5be7c769e0a959ca9"
from {username='slave'} at 127.0.0.1:37730
I20240503 08:09:46.833566 32761 raft_consensus.cc:482] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:46.833585 32761 raft_consensus.cc:3037] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:46.834259 32761 raft_consensus.cc:504] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:46.834455 32761 leader_election.cc:288] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [CANDIDATE]: Term 3 election: Requested vote from peers 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189), 2ac031e1b6da412da3f1c277da38d38b (127.28.55.70:33543)
I20240503 08:09:46.834610 367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5" candidate_uuid: "08561681ad9a4dc5be7c769e0a959ca9" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "2ac031e1b6da412da3f1c277da38d38b"
I20240503 08:09:46.834650 32356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d3a63eb2f640bc9f1ea96a6e071bd5" candidate_uuid: "08561681ad9a4dc5be7c769e0a959ca9" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "441c21339c5c4941943e09a19ca62fe2"
I20240503 08:09:46.834726 32356 raft_consensus.cc:3032] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:09:46.834762 32356 raft_consensus.cc:729] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 441c21339c5c4941943e09a19ca62fe2, State: Running, Role: LEADER
I20240503 08:09:46.834692 367 raft_consensus.cc:3037] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:46.834868 32356 consensus_queue.cc:260] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:46.834954 32356 raft_consensus.cc:3037] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:09:46.835436 367 raft_consensus.cc:2454] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08561681ad9a4dc5be7c769e0a959ca9 in term 3.
I20240503 08:09:46.835454 32356 raft_consensus.cc:2454] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08561681ad9a4dc5be7c769e0a959ca9 in term 3.
I20240503 08:09:46.835645 32725 leader_election.cc:302] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08561681ad9a4dc5be7c769e0a959ca9, 2ac031e1b6da412da3f1c277da38d38b; no voters:
I20240503 08:09:46.835747 895 raft_consensus.cc:2781] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:09:46.835855 895 raft_consensus.cc:686] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEADER]: Becoming Leader. State: Replica: 08561681ad9a4dc5be7c769e0a959ca9, State: Running, Role: LEADER
I20240503 08:09:46.835908 895 consensus_queue.cc:237] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } }
I20240503 08:09:46.836426 32182 catalog_manager.cc:5484] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 reported cstate change: term changed from 2 to 3, leader changed from 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66) to 08561681ad9a4dc5be7c769e0a959ca9 (127.28.55.69). New cstate: current_term: 3 leader_uuid: "08561681ad9a4dc5be7c769e0a959ca9" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "441c21339c5c4941943e09a19ca62fe2" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08561681ad9a4dc5be7c769e0a959ca9" member_type: VOTER last_known_addr { host: "127.28.55.69" port: 45733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac031e1b6da412da3f1c277da38d38b" member_type: VOTER last_known_addr { host: "127.28.55.70" port: 33543 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:196: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240503 08:09:47.368263 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:09:47.371973 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:47.372107 28893 tablet_replica.cc:330] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:47.372176 28893 raft_consensus.cc:2229] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.372223 28893 raft_consensus.cc:2258] T b00c72a8ae02442ca1966c3536976263 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.372411 28893 tablet_replica.cc:330] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:47.372450 28893 raft_consensus.cc:2229] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.372500 28893 raft_consensus.cc:2258] T 2edb5018f69946fdb6ae4a792362ae97 P 94175e832a7f4428be9adc9cfa9032d9 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.372593 28893 tablet_replica.cc:330] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:47.372622 28893 raft_consensus.cc:2229] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.372656 28893 raft_consensus.cc:2258] T 4eeef2123c7341509085d8ca8ff2b2ad P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.372745 28893 tablet_replica.cc:330] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:47.372771 28893 raft_consensus.cc:2229] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.372793 28893 raft_consensus.cc:2258] T df0b30e22bc04f638c852c02354109b6 P 94175e832a7f4428be9adc9cfa9032d9 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.372921 28893 tablet_replica.cc:330] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:47.372951 28893 raft_consensus.cc:2229] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.372982 28893 raft_consensus.cc:2258] T 22779123b80e4ea4a396192e72b73fe8 P 94175e832a7f4428be9adc9cfa9032d9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.373091 28893 tablet_replica.cc:330] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9: stopping tablet replica
I20240503 08:09:47.373129 28893 raft_consensus.cc:2229] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 8 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.373154 28893 raft_consensus.cc:2258] T 10e0365e2b664690ae51f079c4063b8e P 94175e832a7f4428be9adc9cfa9032d9 [term 8 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.385905 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:09:47.387622 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:09:47.391461 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:47.391579 28893 tablet_replica.cc:330] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:47.391633 28893 raft_consensus.cc:2229] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 7 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.391709 28893 raft_consensus.cc:2258] T df0b30e22bc04f638c852c02354109b6 P 441c21339c5c4941943e09a19ca62fe2 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.391889 28893 tablet_replica.cc:330] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:47.391928 28893 raft_consensus.cc:2229] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.391983 28893 raft_consensus.cc:2258] T 7062165e0e3c47c8b43add7f365ec7f4 P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.392102 28893 tablet_replica.cc:330] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:47.392141 28893 raft_consensus.cc:2229] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.392174 28893 raft_consensus.cc:2258] T 22779123b80e4ea4a396192e72b73fe8 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.392288 28893 tablet_replica.cc:330] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:47.392318 28893 raft_consensus.cc:2229] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.392335 28893 raft_consensus.cc:2258] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 441c21339c5c4941943e09a19ca62fe2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.392448 28893 tablet_replica.cc:330] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:47.392478 28893 raft_consensus.cc:2229] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 5 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.392529 28893 raft_consensus.cc:2258] T 4979660e62114184b95e7c2b04a26584 P 441c21339c5c4941943e09a19ca62fe2 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.392645 28893 tablet_replica.cc:330] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2: stopping tablet replica
I20240503 08:09:47.392673 28893 raft_consensus.cc:2229] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.392722 28893 raft_consensus.cc:2258] T 7ac923b6f4774b589c220467ccb7fd8f P 441c21339c5c4941943e09a19ca62fe2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240503 08:09:47.400841 32403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46181: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 08:09:47.401213 32403 consensus_peers.cc:485] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e -> Peer 94175e832a7f4428be9adc9cfa9032d9 (127.28.55.65:46181): Couldn't send request to peer 94175e832a7f4428be9adc9cfa9032d9. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:47.405164 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:09:47.407038 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:09:47.410426 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:47.410560 28893 tablet_replica.cc:330] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:47.410632 28893 raft_consensus.cc:2229] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 8 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.410707 28893 raft_consensus.cc:2258] T 10e0365e2b664690ae51f079c4063b8e P e4df5ad2486247e89c2e44b31357188e [term 8 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.410954 28893 tablet_replica.cc:330] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:47.410998 28893 raft_consensus.cc:2229] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.411059 28893 raft_consensus.cc:2258] T 88518b63f713435f9d72e9cf2a395508 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.411195 28893 tablet_replica.cc:330] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:47.411229 28893 raft_consensus.cc:2229] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.411264 28893 raft_consensus.cc:2258] T df0b30e22bc04f638c852c02354109b6 P e4df5ad2486247e89c2e44b31357188e [term 7 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.411391 28893 tablet_replica.cc:330] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:47.411424 28893 raft_consensus.cc:2229] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.411454 28893 raft_consensus.cc:2258] T 22779123b80e4ea4a396192e72b73fe8 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.411574 28893 tablet_replica.cc:330] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:47.411608 28893 raft_consensus.cc:2229] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.411660 28893 raft_consensus.cc:2258] T b00c72a8ae02442ca1966c3536976263 P e4df5ad2486247e89c2e44b31357188e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.411778 28893 tablet_replica.cc:330] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e: stopping tablet replica
I20240503 08:09:47.411808 28893 raft_consensus.cc:2229] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.411837 28893 raft_consensus.cc:2258] T 4979660e62114184b95e7c2b04a26584 P e4df5ad2486247e89c2e44b31357188e [term 5 FOLLOWER]: Raft consensus is shut down!
W20240503 08:09:47.421383 32724 consensus_peers.cc:485] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 -> Peer 441c21339c5c4941943e09a19ca62fe2 (127.28.55.66:45189): Couldn't send request to peer 441c21339c5c4941943e09a19ca62fe2. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.66:45189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:47.425050 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:09:47.427008 28893 tablet_server.cc:178] TabletServer@127.28.55.68:0 shutting down...
I20240503 08:09:47.430260 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:47.430404 28893 tablet_replica.cc:330] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d: stopping tablet replica
I20240503 08:09:47.430465 28893 raft_consensus.cc:2229] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 6 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.430549 28893 raft_consensus.cc:2258] T 2edb5018f69946fdb6ae4a792362ae97 P 15ce703959504369884c5eac0c7fa77d [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.430845 28893 tablet_replica.cc:330] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d: stopping tablet replica
I20240503 08:09:47.430900 28893 raft_consensus.cc:2229] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.430953 28893 raft_consensus.cc:2258] T 88518b63f713435f9d72e9cf2a395508 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.431077 28893 tablet_replica.cc:330] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d: stopping tablet replica
I20240503 08:09:47.431107 28893 raft_consensus.cc:2229] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.431147 28893 raft_consensus.cc:2258] T b00c72a8ae02442ca1966c3536976263 P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.431260 28893 tablet_replica.cc:330] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d: stopping tablet replica
I20240503 08:09:47.431289 28893 raft_consensus.cc:2229] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 5 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.431324 28893 raft_consensus.cc:2258] T 4979660e62114184b95e7c2b04a26584 P 15ce703959504369884c5eac0c7fa77d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.431432 28893 tablet_replica.cc:330] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d: stopping tablet replica
I20240503 08:09:47.431459 28893 raft_consensus.cc:2229] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.431511 28893 raft_consensus.cc:2258] T 55c3a314149b4921b6cddce536a010e9 P 15ce703959504369884c5eac0c7fa77d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.431597 28893 tablet_replica.cc:330] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d: stopping tablet replica
I20240503 08:09:47.431630 28893 raft_consensus.cc:2229] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.431679 28893 raft_consensus.cc:2258] T 4eeef2123c7341509085d8ca8ff2b2ad P 15ce703959504369884c5eac0c7fa77d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.443559 28893 tablet_server.cc:194] TabletServer@127.28.55.68:0 shutdown complete.
I20240503 08:09:47.444993 28893 tablet_server.cc:178] TabletServer@127.28.55.69:0 shutting down...
I20240503 08:09:47.448236 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:47.448382 28893 tablet_replica.cc:330] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9: stopping tablet replica
I20240503 08:09:47.448465 28893 raft_consensus.cc:2229] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.448534 28893 raft_consensus.cc:2258] T 4eeef2123c7341509085d8ca8ff2b2ad P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.448695 28893 tablet_replica.cc:330] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9: stopping tablet replica
I20240503 08:09:47.448729 28893 raft_consensus.cc:2229] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.448761 28893 raft_consensus.cc:2258] T 7ac923b6f4774b589c220467ccb7fd8f P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.448907 28893 tablet_replica.cc:330] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9: stopping tablet replica
I20240503 08:09:47.448941 28893 raft_consensus.cc:2229] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 6 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.448992 28893 raft_consensus.cc:2258] T 2edb5018f69946fdb6ae4a792362ae97 P 08561681ad9a4dc5be7c769e0a959ca9 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.449072 28893 tablet_replica.cc:330] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9: stopping tablet replica
I20240503 08:09:47.449101 28893 raft_consensus.cc:2229] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.449149 28893 raft_consensus.cc:2258] T 7062165e0e3c47c8b43add7f365ec7f4 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.449241 28893 tablet_replica.cc:330] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9: stopping tablet replica
I20240503 08:09:47.449266 28893 raft_consensus.cc:2229] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.449314 28893 raft_consensus.cc:2258] T 55c3a314149b4921b6cddce536a010e9 P 08561681ad9a4dc5be7c769e0a959ca9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.449410 28893 tablet_replica.cc:330] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: stopping tablet replica
I20240503 08:09:47.449436 28893 raft_consensus.cc:2229] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.449501 28893 pending_rounds.cc:62] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: Trying to abort 1 pending ops.
I20240503 08:09:47.449519 28893 pending_rounds.cc:68] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9: Aborting op as it isn't in flight: id { term: 3 index: 32 } timestamp: 7023508630880509952 op_type: NO_OP noop_request { }
I20240503 08:09:47.449604 28893 raft_consensus.cc:2866] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:09:47.449620 28893 raft_consensus.cc:2258] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 08561681ad9a4dc5be7c769e0a959ca9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.462684 28893 tablet_server.cc:194] TabletServer@127.28.55.69:0 shutdown complete.
I20240503 08:09:47.464241 28893 tablet_server.cc:178] TabletServer@127.28.55.70:0 shutting down...
I20240503 08:09:47.467257 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:47.467378 28893 tablet_replica.cc:330] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b: stopping tablet replica
I20240503 08:09:47.467471 28893 raft_consensus.cc:2229] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.467532 28893 raft_consensus.cc:2258] T 55c3a314149b4921b6cddce536a010e9 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.467700 28893 tablet_replica.cc:330] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b: stopping tablet replica
I20240503 08:09:47.467733 28893 raft_consensus.cc:2229] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.467770 28893 raft_consensus.cc:2258] T 07d3a63eb2f640bc9f1ea96a6e071bd5 P 2ac031e1b6da412da3f1c277da38d38b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.467859 28893 tablet_replica.cc:330] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b: stopping tablet replica
I20240503 08:09:47.467890 28893 raft_consensus.cc:2229] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 8 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.467917 28893 raft_consensus.cc:2258] T 10e0365e2b664690ae51f079c4063b8e P 2ac031e1b6da412da3f1c277da38d38b [term 8 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.467994 28893 tablet_replica.cc:330] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b: stopping tablet replica
I20240503 08:09:47.468021 28893 raft_consensus.cc:2229] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:47.468052 28893 raft_consensus.cc:2258] T 7062165e0e3c47c8b43add7f365ec7f4 P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.468128 28893 tablet_replica.cc:330] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b: stopping tablet replica
I20240503 08:09:47.468153 28893 raft_consensus.cc:2229] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.468200 28893 raft_consensus.cc:2258] T 7ac923b6f4774b589c220467ccb7fd8f P 2ac031e1b6da412da3f1c277da38d38b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.468282 28893 tablet_replica.cc:330] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b: stopping tablet replica
I20240503 08:09:47.468307 28893 raft_consensus.cc:2229] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.468350 28893 raft_consensus.cc:2258] T 88518b63f713435f9d72e9cf2a395508 P 2ac031e1b6da412da3f1c277da38d38b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.480835 28893 tablet_server.cc:194] TabletServer@127.28.55.70:0 shutdown complete.
I20240503 08:09:47.482327 28893 master.cc:537] Master@127.28.55.126:46853 shutting down...
I20240503 08:09:47.486059 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:47.486145 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:47.486163 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2b393cef270e43ffbd065f91d7add5ed: stopping tablet replica
I20240503 08:09:47.498241 28893 master.cc:552] Master@127.28.55.126:46853 shutdown complete.
I20240503 08:09:47.500509 28893 test_util.cc:175] -----------------------------------------------
I20240503 08:09:47.500561 28893 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714723660874102-28893-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (34879 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240503 08:09:47.501062 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:46053
I20240503 08:09:47.501205 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:47.502683 1005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:47.502694 1007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:47.502921 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:47.502748 1004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:47.503115 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:47.503144 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:47.503159 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723787503159 us; error 0 us; skew 500 ppm
I20240503 08:09:47.508399 28893 webserver.cc:460] Webserver started at http://127.28.55.126:35637/ using document root <none> and password file <none>
I20240503 08:09:47.508496 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:47.508548 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:47.508597 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:47.508890 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "3320b646552142ea8f51cedce3652e5d"
format_stamp: "Formatted at 2024-05-03 08:09:47 on dist-test-slave-mlgg"
I20240503 08:09:47.509717 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:47.510161 1012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.510295 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.510336 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "3320b646552142ea8f51cedce3652e5d"
format_stamp: "Formatted at 2024-05-03 08:09:47 on dist-test-slave-mlgg"
I20240503 08:09:47.510381 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:47.520572 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:47.520769 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:47.523278 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:46053
I20240503 08:09:47.529480 1064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:46053 every 8 connection(s)
I20240503 08:09:47.529592 1065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:47.530691 1065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d: Bootstrap starting.
I20240503 08:09:47.531023 1065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:47.531548 1065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d: No bootstrap required, opened a new log
I20240503 08:09:47.531668 1065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3320b646552142ea8f51cedce3652e5d" member_type: VOTER }
I20240503 08:09:47.531726 1065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:47.531737 1065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3320b646552142ea8f51cedce3652e5d, State: Initialized, Role: FOLLOWER
I20240503 08:09:47.531783 1065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3320b646552142ea8f51cedce3652e5d" member_type: VOTER }
I20240503 08:09:47.531821 1065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:09:47.531836 1065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:09:47.531860 1065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:47.532379 1065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3320b646552142ea8f51cedce3652e5d" member_type: VOTER }
I20240503 08:09:47.532436 1065 leader_election.cc:302] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3320b646552142ea8f51cedce3652e5d; no voters:
I20240503 08:09:47.532529 1065 leader_election.cc:288] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:09:47.532593 1068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:47.532692 1065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:09:47.532765 1068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 1 LEADER]: Becoming Leader. State: Replica: 3320b646552142ea8f51cedce3652e5d, State: Running, Role: LEADER
I20240503 08:09:47.532825 1068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3320b646552142ea8f51cedce3652e5d" member_type: VOTER }
I20240503 08:09:47.533066 1069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3320b646552142ea8f51cedce3652e5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3320b646552142ea8f51cedce3652e5d" member_type: VOTER } }
I20240503 08:09:47.533066 1070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3320b646552142ea8f51cedce3652e5d. Latest consensus state: current_term: 1 leader_uuid: "3320b646552142ea8f51cedce3652e5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3320b646552142ea8f51cedce3652e5d" member_type: VOTER } }
I20240503 08:09:47.533152 1069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:47.533159 1070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:47.533466 1075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:09:47.533612 1075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:09:47.533939 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:09:47.534121 1075 catalog_manager.cc:1347] Generated new cluster ID: 07704e0a9f4a44f99f0dddcd52bff9ca
I20240503 08:09:47.534168 1075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:09:47.540856 1075 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:09:47.541312 1075 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:09:47.547077 1075 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d: Generated new TSK 0
I20240503 08:09:47.547173 1075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:09:47.549675 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:09:47.551263 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:47.551329 1086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:47.551398 1089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:47.551409 1087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:47.551643 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:47.551674 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:47.551685 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723787551686 us; error 0 us; skew 500 ppm
I20240503 08:09:47.552553 28893 webserver.cc:460] Webserver started at http://127.28.55.65:34125/ using document root <none> and password file <none>
I20240503 08:09:47.552646 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:47.552699 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:47.552752 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:47.553097 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "41c5a42d9b8242f3a9aa4ee972b6e850"
format_stamp: "Formatted at 2024-05-03 08:09:47 on dist-test-slave-mlgg"
I20240503 08:09:47.554183 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:47.554656 1094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.554772 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.554852 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "41c5a42d9b8242f3a9aa4ee972b6e850"
format_stamp: "Formatted at 2024-05-03 08:09:47 on dist-test-slave-mlgg"
I20240503 08:09:47.554901 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:47.562520 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:47.562702 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:47.562893 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:47.563076 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:47.563100 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.563125 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:47.563140 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.565717 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:44877
I20240503 08:09:47.565773 1157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:44877 every 8 connection(s)
I20240503 08:09:47.566099 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:47.567260 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:47.567288 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:47.567327 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:47.567603 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:47.567700 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:47.567740 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:47.567752 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723787567753 us; error 0 us; skew 500 ppm
I20240503 08:09:47.568468 28893 webserver.cc:460] Webserver started at http://127.28.55.66:33349/ using document root <none> and password file <none>
I20240503 08:09:47.568555 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:47.568601 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:47.568673 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:47.568953 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "e8406787837241b0a70b00e5943ab1dc"
format_stamp: "Formatted at 2024-05-03 08:09:47 on dist-test-slave-mlgg"
I20240503 08:09:47.569547 1158 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46053
I20240503 08:09:47.569609 1158 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:47.569695 1158 heartbeater.cc:511] Master 127.28.55.126:46053 requested a full tablet report, sending...
I20240503 08:09:47.569919 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:47.569933 1029 ts_manager.cc:194] Registered new tserver with Master: 41c5a42d9b8242f3a9aa4ee972b6e850 (127.28.55.65:44877)
I20240503 08:09:47.570400 1170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.570529 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.570534 1029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47574
I20240503 08:09:47.570582 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "e8406787837241b0a70b00e5943ab1dc"
format_stamp: "Formatted at 2024-05-03 08:09:47 on dist-test-slave-mlgg"
I20240503 08:09:47.570631 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:47.578114 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:47.578323 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:47.578461 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:47.578667 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:47.578694 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.578722 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:47.578745 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.581350 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:44827
I20240503 08:09:47.581421 1233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:44827 every 8 connection(s)
I20240503 08:09:47.581722 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:47.583060 1237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:47.583209 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:47.583067 1238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:47.583128 1240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:47.583746 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:47.583781 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:47.583796 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723787583797 us; error 0 us; skew 500 ppm
I20240503 08:09:47.584600 28893 webserver.cc:460] Webserver started at http://127.28.55.67:42201/ using document root <none> and password file <none>
I20240503 08:09:47.584690 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:47.584739 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:47.584789 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:47.585136 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "ceabd150b5d941f79238f0b7bff8a410"
format_stamp: "Formatted at 2024-05-03 08:09:47 on dist-test-slave-mlgg"
I20240503 08:09:47.585181 1234 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46053
I20240503 08:09:47.585228 1234 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:47.585361 1234 heartbeater.cc:511] Master 127.28.55.126:46053 requested a full tablet report, sending...
I20240503 08:09:47.585600 1029 ts_manager.cc:194] Registered new tserver with Master: e8406787837241b0a70b00e5943ab1dc (127.28.55.66:44827)
I20240503 08:09:47.586078 1029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47584
I20240503 08:09:47.586243 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:47.586778 1245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.586951 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.586992 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "ceabd150b5d941f79238f0b7bff8a410"
format_stamp: "Formatted at 2024-05-03 08:09:47 on dist-test-slave-mlgg"
I20240503 08:09:47.587041 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:47.594292 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:47.594569 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:47.594725 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:47.594980 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:47.595008 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.595036 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:47.595060 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:47.597736 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:34171
I20240503 08:09:47.597810 1308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:34171 every 8 connection(s)
I20240503 08:09:47.600980 1309 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46053
I20240503 08:09:47.601049 1309 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:47.601152 1309 heartbeater.cc:511] Master 127.28.55.126:46053 requested a full tablet report, sending...
I20240503 08:09:47.601380 1029 ts_manager.cc:194] Registered new tserver with Master: ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67:34171)
I20240503 08:09:47.601702 1029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47596
I20240503 08:09:47.602340 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004434547s
I20240503 08:09:48.571143 1158 heartbeater.cc:503] Master 127.28.55.126:46053 was elected leader, sending a full tablet report...
I20240503 08:09:48.586664 1234 heartbeater.cc:503] Master 127.28.55.126:46053 was elected leader, sending a full tablet report...
I20240503 08:09:48.602298 1309 heartbeater.cc:503] Master 127.28.55.126:46053 was elected leader, sending a full tablet report...
I20240503 08:09:48.627595 28893 test_util.cc:268] Using random seed: 1045372550
I20240503 08:09:48.633201 1029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45888:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:09:48.633368 1029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:09:48.638588 1121 tablet_service.cc:1467] Processing CreateTablet for tablet c63aa1b4f3f04c50abb4c4871f64b045 (DEFAULT_TABLE table=test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 08:09:48.638588 1122 tablet_service.cc:1467] Processing CreateTablet for tablet f15826aece5b461b81ae8a5629b02ebb (DEFAULT_TABLE table=test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 08:09:48.638774 1121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63aa1b4f3f04c50abb4c4871f64b045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:48.638888 1198 tablet_service.cc:1467] Processing CreateTablet for tablet f15826aece5b461b81ae8a5629b02ebb (DEFAULT_TABLE table=test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 08:09:48.638969 1198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15826aece5b461b81ae8a5629b02ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:48.640158 1196 tablet_service.cc:1467] Processing CreateTablet for tablet 3e756374f39045bf8e444601b2b53d72 (DEFAULT_TABLE table=test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 08:09:48.640197 1330 tablet_bootstrap.cc:492] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850: Bootstrap starting.
I20240503 08:09:48.640213 1120 tablet_service.cc:1467] Processing CreateTablet for tablet 3e756374f39045bf8e444601b2b53d72 (DEFAULT_TABLE table=test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 08:09:48.640267 1196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e756374f39045bf8e444601b2b53d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:48.640321 1120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e756374f39045bf8e444601b2b53d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:48.640642 1330 tablet_bootstrap.cc:654] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:48.640902 1273 tablet_service.cc:1467] Processing CreateTablet for tablet f15826aece5b461b81ae8a5629b02ebb (DEFAULT_TABLE table=test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 08:09:48.641005 1273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15826aece5b461b81ae8a5629b02ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:48.641479 1197 tablet_service.cc:1467] Processing CreateTablet for tablet c63aa1b4f3f04c50abb4c4871f64b045 (DEFAULT_TABLE table=test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 08:09:48.641542 1272 tablet_service.cc:1467] Processing CreateTablet for tablet c63aa1b4f3f04c50abb4c4871f64b045 (DEFAULT_TABLE table=test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 08:09:48.641569 1197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63aa1b4f3f04c50abb4c4871f64b045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:48.641610 1272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63aa1b4f3f04c50abb4c4871f64b045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:48.642036 1332 tablet_bootstrap.cc:492] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410: Bootstrap starting.
I20240503 08:09:48.642328 1332 tablet_bootstrap.cc:654] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:48.642693 1271 tablet_service.cc:1467] Processing CreateTablet for tablet 3e756374f39045bf8e444601b2b53d72 (DEFAULT_TABLE table=test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 08:09:48.642787 1271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e756374f39045bf8e444601b2b53d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:48.642864 1332 tablet_bootstrap.cc:492] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410: No bootstrap required, opened a new log
I20240503 08:09:48.642905 1332 ts_tablet_manager.cc:1367] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:48.640218 1329 tablet_bootstrap.cc:492] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc: Bootstrap starting.
I20240503 08:09:48.643062 1332 raft_consensus.cc:348] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.643128 1332 raft_consensus.cc:374] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:48.643146 1332 raft_consensus.cc:729] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceabd150b5d941f79238f0b7bff8a410, State: Initialized, Role: FOLLOWER
I20240503 08:09:48.643203 1332 consensus_queue.cc:260] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.643258 1122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15826aece5b461b81ae8a5629b02ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:48.643329 1329 tablet_bootstrap.cc:654] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:48.643348 1332 ts_tablet_manager.cc:1398] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.643743 1332 tablet_bootstrap.cc:492] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410: Bootstrap starting.
I20240503 08:09:48.644079 1332 tablet_bootstrap.cc:654] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:48.644690 1332 tablet_bootstrap.cc:492] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410: No bootstrap required, opened a new log
I20240503 08:09:48.644737 1332 ts_tablet_manager.cc:1367] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:48.644743 1330 tablet_bootstrap.cc:492] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850: No bootstrap required, opened a new log
I20240503 08:09:48.644743 1329 tablet_bootstrap.cc:492] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc: No bootstrap required, opened a new log
I20240503 08:09:48.644832 1330 ts_tablet_manager.cc:1367] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:09:48.644834 1329 ts_tablet_manager.cc:1367] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:09:48.644897 1332 raft_consensus.cc:348] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.644958 1332 raft_consensus.cc:374] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:48.644966 1330 raft_consensus.cc:348] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.644975 1329 raft_consensus.cc:348] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.645020 1330 raft_consensus.cc:374] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:48.645092 1329 raft_consensus.cc:374] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:48.644975 1332 raft_consensus.cc:729] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceabd150b5d941f79238f0b7bff8a410, State: Initialized, Role: FOLLOWER
I20240503 08:09:48.645110 1330 raft_consensus.cc:729] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c5a42d9b8242f3a9aa4ee972b6e850, State: Initialized, Role: FOLLOWER
I20240503 08:09:48.645112 1329 raft_consensus.cc:729] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406787837241b0a70b00e5943ab1dc, State: Initialized, Role: FOLLOWER
I20240503 08:09:48.645161 1330 consensus_queue.cc:260] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.645164 1332 consensus_queue.cc:260] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.645195 1329 consensus_queue.cc:260] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.645278 1332 ts_tablet_manager.cc:1398] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.645308 1330 ts_tablet_manager.cc:1398] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.645334 1329 ts_tablet_manager.cc:1398] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.645337 1332 tablet_bootstrap.cc:492] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410: Bootstrap starting.
I20240503 08:09:48.645406 1329 tablet_bootstrap.cc:492] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc: Bootstrap starting.
I20240503 08:09:48.645761 1332 tablet_bootstrap.cc:654] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:48.645812 1329 tablet_bootstrap.cc:654] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:48.645361 1330 tablet_bootstrap.cc:492] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850: Bootstrap starting.
I20240503 08:09:48.646407 1329 tablet_bootstrap.cc:492] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc: No bootstrap required, opened a new log
I20240503 08:09:48.646448 1329 ts_tablet_manager.cc:1367] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:48.646574 1329 raft_consensus.cc:348] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.646624 1329 raft_consensus.cc:374] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:48.646642 1329 raft_consensus.cc:729] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406787837241b0a70b00e5943ab1dc, State: Initialized, Role: FOLLOWER
I20240503 08:09:48.646713 1329 consensus_queue.cc:260] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.646646 1330 tablet_bootstrap.cc:654] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:48.646798 1329 ts_tablet_manager.cc:1398] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.646911 1329 tablet_bootstrap.cc:492] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc: Bootstrap starting.
I20240503 08:09:48.647295 1329 tablet_bootstrap.cc:654] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:48.647377 1330 tablet_bootstrap.cc:492] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850: No bootstrap required, opened a new log
I20240503 08:09:48.647403 1332 tablet_bootstrap.cc:492] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410: No bootstrap required, opened a new log
I20240503 08:09:48.647426 1330 ts_tablet_manager.cc:1367] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:48.647445 1332 ts_tablet_manager.cc:1367] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:09:48.647612 1330 raft_consensus.cc:348] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.647678 1330 raft_consensus.cc:374] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:48.647617 1332 raft_consensus.cc:348] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.647753 1332 raft_consensus.cc:374] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:48.647769 1332 raft_consensus.cc:729] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceabd150b5d941f79238f0b7bff8a410, State: Initialized, Role: FOLLOWER
I20240503 08:09:48.647822 1332 consensus_queue.cc:260] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.647910 1332 ts_tablet_manager.cc:1398] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.647897 1329 tablet_bootstrap.cc:492] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc: No bootstrap required, opened a new log
I20240503 08:09:48.648011 1329 ts_tablet_manager.cc:1367] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:48.647698 1330 raft_consensus.cc:729] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c5a42d9b8242f3a9aa4ee972b6e850, State: Initialized, Role: FOLLOWER
I20240503 08:09:48.648141 1330 consensus_queue.cc:260] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.648187 1329 raft_consensus.cc:348] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.648242 1329 raft_consensus.cc:374] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:48.648252 1330 ts_tablet_manager.cc:1398] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:48.648262 1329 raft_consensus.cc:729] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406787837241b0a70b00e5943ab1dc, State: Initialized, Role: FOLLOWER
I20240503 08:09:48.648331 1329 consensus_queue.cc:260] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.648428 1329 ts_tablet_manager.cc:1398] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.648567 1330 tablet_bootstrap.cc:492] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850: Bootstrap starting.
I20240503 08:09:48.648886 1330 tablet_bootstrap.cc:654] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:48.649498 1330 tablet_bootstrap.cc:492] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850: No bootstrap required, opened a new log
I20240503 08:09:48.649545 1330 ts_tablet_manager.cc:1367] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:48.649679 1330 raft_consensus.cc:348] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.649739 1330 raft_consensus.cc:374] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:48.649757 1330 raft_consensus.cc:729] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c5a42d9b8242f3a9aa4ee972b6e850, State: Initialized, Role: FOLLOWER
I20240503 08:09:48.649798 1330 consensus_queue.cc:260] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.649890 1330 ts_tablet_manager.cc:1398] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.656108 1338 raft_consensus.cc:482] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:48.656196 1338 raft_consensus.cc:504] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.656349 1338 leader_election.cc:288] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c5a42d9b8242f3a9aa4ee972b6e850 (127.28.55.65:44877), ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67:34171)
I20240503 08:09:48.659066 1283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e756374f39045bf8e444601b2b53d72" candidate_uuid: "e8406787837241b0a70b00e5943ab1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceabd150b5d941f79238f0b7bff8a410" is_pre_election: true
I20240503 08:09:48.659130 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e756374f39045bf8e444601b2b53d72" candidate_uuid: "e8406787837241b0a70b00e5943ab1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" is_pre_election: true
I20240503 08:09:48.659193 1132 raft_consensus.cc:2454] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406787837241b0a70b00e5943ab1dc in term 0.
I20240503 08:09:48.659191 1283 raft_consensus.cc:2454] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406787837241b0a70b00e5943ab1dc in term 0.
I20240503 08:09:48.659368 1173 leader_election.cc:302] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceabd150b5d941f79238f0b7bff8a410, e8406787837241b0a70b00e5943ab1dc; no voters:
I20240503 08:09:48.659474 1338 raft_consensus.cc:2781] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:48.659528 1338 raft_consensus.cc:482] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:48.659549 1338 raft_consensus.cc:3037] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:48.660202 1338 raft_consensus.cc:504] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.660324 1338 leader_election.cc:288] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [CANDIDATE]: Term 1 election: Requested vote from peers 41c5a42d9b8242f3a9aa4ee972b6e850 (127.28.55.65:44877), ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67:34171)
I20240503 08:09:48.660463 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e756374f39045bf8e444601b2b53d72" candidate_uuid: "e8406787837241b0a70b00e5943ab1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850"
I20240503 08:09:48.660490 1283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e756374f39045bf8e444601b2b53d72" candidate_uuid: "e8406787837241b0a70b00e5943ab1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceabd150b5d941f79238f0b7bff8a410"
I20240503 08:09:48.660567 1132 raft_consensus.cc:3037] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:48.660583 1283 raft_consensus.cc:3037] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:48.661320 1283 raft_consensus.cc:2454] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406787837241b0a70b00e5943ab1dc in term 1.
I20240503 08:09:48.661320 1132 raft_consensus.cc:2454] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406787837241b0a70b00e5943ab1dc in term 1.
I20240503 08:09:48.661543 1173 leader_election.cc:302] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c5a42d9b8242f3a9aa4ee972b6e850, e8406787837241b0a70b00e5943ab1dc; no voters:
I20240503 08:09:48.661623 1338 raft_consensus.cc:2781] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:48.661763 1338 raft_consensus.cc:686] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Becoming Leader. State: Replica: e8406787837241b0a70b00e5943ab1dc, State: Running, Role: LEADER
I20240503 08:09:48.661820 1338 consensus_queue.cc:237] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.662477 1029 catalog_manager.cc:5484] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc reported cstate change: term changed from 0 to 1, leader changed from <none> to e8406787837241b0a70b00e5943ab1dc (127.28.55.66). New cstate: current_term: 1 leader_uuid: "e8406787837241b0a70b00e5943ab1dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:48.664391 1334 raft_consensus.cc:482] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:48.664448 1334 raft_consensus.cc:504] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.664605 1334 leader_election.cc:288] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c5a42d9b8242f3a9aa4ee972b6e850 (127.28.55.65:44877), e8406787837241b0a70b00e5943ab1dc (127.28.55.66:44827)
I20240503 08:09:48.667964 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63aa1b4f3f04c50abb4c4871f64b045" candidate_uuid: "ceabd150b5d941f79238f0b7bff8a410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" is_pre_election: true
I20240503 08:09:48.668048 1132 raft_consensus.cc:2454] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceabd150b5d941f79238f0b7bff8a410 in term 0.
I20240503 08:09:48.668030 1208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63aa1b4f3f04c50abb4c4871f64b045" candidate_uuid: "ceabd150b5d941f79238f0b7bff8a410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406787837241b0a70b00e5943ab1dc" is_pre_election: true
I20240503 08:09:48.668118 1208 raft_consensus.cc:2454] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceabd150b5d941f79238f0b7bff8a410 in term 0.
I20240503 08:09:48.668226 1248 leader_election.cc:302] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c5a42d9b8242f3a9aa4ee972b6e850, ceabd150b5d941f79238f0b7bff8a410; no voters:
I20240503 08:09:48.668311 1334 raft_consensus.cc:2781] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:48.668352 1334 raft_consensus.cc:482] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:48.668365 1334 raft_consensus.cc:3037] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:48.669041 1334 raft_consensus.cc:504] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.669152 1334 leader_election.cc:288] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [CANDIDATE]: Term 1 election: Requested vote from peers 41c5a42d9b8242f3a9aa4ee972b6e850 (127.28.55.65:44877), e8406787837241b0a70b00e5943ab1dc (127.28.55.66:44827)
I20240503 08:09:48.669344 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63aa1b4f3f04c50abb4c4871f64b045" candidate_uuid: "ceabd150b5d941f79238f0b7bff8a410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850"
I20240503 08:09:48.669427 1132 raft_consensus.cc:3037] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:48.669452 1208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63aa1b4f3f04c50abb4c4871f64b045" candidate_uuid: "ceabd150b5d941f79238f0b7bff8a410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406787837241b0a70b00e5943ab1dc"
I20240503 08:09:48.669525 1208 raft_consensus.cc:3037] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:48.670001 1132 raft_consensus.cc:2454] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceabd150b5d941f79238f0b7bff8a410 in term 1.
I20240503 08:09:48.670106 1208 raft_consensus.cc:2454] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceabd150b5d941f79238f0b7bff8a410 in term 1.
I20240503 08:09:48.670229 1248 leader_election.cc:302] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c5a42d9b8242f3a9aa4ee972b6e850, ceabd150b5d941f79238f0b7bff8a410; no voters:
I20240503 08:09:48.670305 1334 raft_consensus.cc:2781] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:48.670452 1334 raft_consensus.cc:686] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 1 LEADER]: Becoming Leader. State: Replica: ceabd150b5d941f79238f0b7bff8a410, State: Running, Role: LEADER
I20240503 08:09:48.670496 1334 consensus_queue.cc:237] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.671111 1029 catalog_manager.cc:5484] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "ceabd150b5d941f79238f0b7bff8a410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:48.719009 1338 raft_consensus.cc:482] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:48.719130 1338 raft_consensus.cc:504] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.719302 1338 leader_election.cc:288] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c5a42d9b8242f3a9aa4ee972b6e850 (127.28.55.65:44877), ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67:34171)
I20240503 08:09:48.719525 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15826aece5b461b81ae8a5629b02ebb" candidate_uuid: "e8406787837241b0a70b00e5943ab1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" is_pre_election: true
I20240503 08:09:48.719611 1132 raft_consensus.cc:2454] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406787837241b0a70b00e5943ab1dc in term 0.
I20240503 08:09:48.719585 1283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15826aece5b461b81ae8a5629b02ebb" candidate_uuid: "e8406787837241b0a70b00e5943ab1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceabd150b5d941f79238f0b7bff8a410" is_pre_election: true
I20240503 08:09:48.719666 1283 raft_consensus.cc:2454] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406787837241b0a70b00e5943ab1dc in term 0.
I20240503 08:09:48.719748 1173 leader_election.cc:302] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c5a42d9b8242f3a9aa4ee972b6e850, e8406787837241b0a70b00e5943ab1dc; no voters:
I20240503 08:09:48.719841 1338 raft_consensus.cc:2781] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:48.719880 1338 raft_consensus.cc:482] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:48.719893 1338 raft_consensus.cc:3037] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:48.720546 1338 raft_consensus.cc:504] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.720649 1338 leader_election.cc:288] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [CANDIDATE]: Term 1 election: Requested vote from peers 41c5a42d9b8242f3a9aa4ee972b6e850 (127.28.55.65:44877), ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67:34171)
I20240503 08:09:48.720824 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15826aece5b461b81ae8a5629b02ebb" candidate_uuid: "e8406787837241b0a70b00e5943ab1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850"
I20240503 08:09:48.720889 1132 raft_consensus.cc:3037] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:48.720865 1283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15826aece5b461b81ae8a5629b02ebb" candidate_uuid: "e8406787837241b0a70b00e5943ab1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceabd150b5d941f79238f0b7bff8a410"
I20240503 08:09:48.720918 1283 raft_consensus.cc:3037] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:48.721443 1132 raft_consensus.cc:2454] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406787837241b0a70b00e5943ab1dc in term 1.
I20240503 08:09:48.721460 1283 raft_consensus.cc:2454] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406787837241b0a70b00e5943ab1dc in term 1.
I20240503 08:09:48.721609 1173 leader_election.cc:302] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c5a42d9b8242f3a9aa4ee972b6e850, e8406787837241b0a70b00e5943ab1dc; no voters:
I20240503 08:09:48.721688 1338 raft_consensus.cc:2781] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:48.721742 1338 raft_consensus.cc:686] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Becoming Leader. State: Replica: e8406787837241b0a70b00e5943ab1dc, State: Running, Role: LEADER
I20240503 08:09:48.721794 1338 consensus_queue.cc:237] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:48.722437 1029 catalog_manager.cc:5484] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc reported cstate change: term changed from 0 to 1, leader changed from <none> to e8406787837241b0a70b00e5943ab1dc (127.28.55.66). New cstate: current_term: 1 leader_uuid: "e8406787837241b0a70b00e5943ab1dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:48.730424 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:48.731853 1356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:48.731946 1354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:48.731959 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:48.732146 1353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:48.732301 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:48.732349 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:48.732362 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723788732362 us; error 0 us; skew 500 ppm
I20240503 08:09:48.733098 28893 webserver.cc:460] Webserver started at http://127.28.55.68:40283/ using document root <none> and password file <none>
I20240503 08:09:48.733196 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:48.733251 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:48.733297 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:48.733606 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-3-root/instance:
uuid: "28260ffb214e412794ff375d47947ad9"
format_stamp: "Formatted at 2024-05-03 08:09:48 on dist-test-slave-mlgg"
I20240503 08:09:48.734470 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:48.734923 1361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.735073 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.735116 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-3-root
uuid: "28260ffb214e412794ff375d47947ad9"
format_stamp: "Formatted at 2024-05-03 08:09:48 on dist-test-slave-mlgg"
I20240503 08:09:48.735162 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:48.745133 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:48.745400 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:48.745573 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:48.745791 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:48.745818 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.745846 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:48.745870 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:48.749253 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.68:35335
I20240503 08:09:48.749297 1424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.68:35335 every 8 connection(s)
I20240503 08:09:48.752650 1425 heartbeater.cc:347] Connected to a master server at 127.28.55.126:46053
I20240503 08:09:48.752725 1425 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:48.752869 1425 heartbeater.cc:511] Master 127.28.55.126:46053 requested a full tablet report, sending...
I20240503 08:09:48.753130 1029 ts_manager.cc:194] Registered new tserver with Master: 28260ffb214e412794ff375d47947ad9 (127.28.55.68:35335)
I20240503 08:09:48.753515 1029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45892
I20240503 08:09:48.762495 1338 consensus_queue.cc:1035] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:09:48.763571 1346 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240503 08:09:48.763617 1346 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240503 08:09:48.765658 1334 consensus_queue.cc:1035] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:48.767508 1338 consensus_queue.cc:1035] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:48.785734 1334 consensus_queue.cc:1035] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:48.798243 1338 consensus_queue.cc:1035] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:09:48.816685 1159 tablet.cc:2369] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:09:48.820722 1338 consensus_queue.cc:1035] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:49.534067 1083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240503 08:09:49.534360 1078 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table dfd70180e3bb4b82a928387473cfbcbf: could not find any suitable replica to move from server e8406787837241b0a70b00e5943ab1dc to server 28260ffb214e412794ff375d47947ad9
I20240503 08:09:49.537499 1208 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e756374f39045bf8e444601b2b53d72"
dest_uuid: "e8406787837241b0a70b00e5943ab1dc"
mode: GRACEFUL
new_leader_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850"
from {username='slave'} at 127.0.0.1:40382
I20240503 08:09:49.537603 1208 raft_consensus.cc:595] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Received request to transfer leadership to TS 41c5a42d9b8242f3a9aa4ee972b6e850
I20240503 08:09:49.537889 1083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 08:09:49.537954 1083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:49.667856 1439 raft_consensus.cc:979] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc: : Instructing follower 41c5a42d9b8242f3a9aa4ee972b6e850 to start an election
I20240503 08:09:49.667949 1439 raft_consensus.cc:1067] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Signalling peer 41c5a42d9b8242f3a9aa4ee972b6e850 to start an election
I20240503 08:09:49.668167 1132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e756374f39045bf8e444601b2b53d72"
dest_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850"
from {username='slave'} at 127.0.0.1:56230
I20240503 08:09:49.668239 1132 raft_consensus.cc:482] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:49.668265 1132 raft_consensus.cc:3037] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:49.668910 1132 raft_consensus.cc:504] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:49.669063 1132 leader_election.cc:288] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [CANDIDATE]: Term 2 election: Requested vote from peers e8406787837241b0a70b00e5943ab1dc (127.28.55.66:44827), ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67:34171)
I20240503 08:09:49.671880 1283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e756374f39045bf8e444601b2b53d72" candidate_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ceabd150b5d941f79238f0b7bff8a410"
I20240503 08:09:49.671967 1283 raft_consensus.cc:3037] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:49.671945 1208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e756374f39045bf8e444601b2b53d72" candidate_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e8406787837241b0a70b00e5943ab1dc"
I20240503 08:09:49.672065 1208 raft_consensus.cc:3032] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:49.672097 1208 raft_consensus.cc:729] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8406787837241b0a70b00e5943ab1dc, State: Running, Role: LEADER
I20240503 08:09:49.672152 1208 consensus_queue.cc:260] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:49.672233 1208 raft_consensus.cc:3037] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:49.672700 1283 raft_consensus.cc:2454] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c5a42d9b8242f3a9aa4ee972b6e850 in term 2.
I20240503 08:09:49.672816 1208 raft_consensus.cc:2454] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c5a42d9b8242f3a9aa4ee972b6e850 in term 2.
I20240503 08:09:49.672884 1097 leader_election.cc:302] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c5a42d9b8242f3a9aa4ee972b6e850, ceabd150b5d941f79238f0b7bff8a410; no voters:
I20240503 08:09:49.673077 1445 raft_consensus.cc:2781] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:49.673223 1445 raft_consensus.cc:686] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 2 LEADER]: Becoming Leader. State: Replica: 41c5a42d9b8242f3a9aa4ee972b6e850, State: Running, Role: LEADER
I20240503 08:09:49.673278 1445 consensus_queue.cc:237] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } }
I20240503 08:09:49.673714 1029 catalog_manager.cc:5484] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 reported cstate change: term changed from 1 to 2, leader changed from e8406787837241b0a70b00e5943ab1dc (127.28.55.66) to 41c5a42d9b8242f3a9aa4ee972b6e850 (127.28.55.65). New cstate: current_term: 2 leader_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:49.754122 1425 heartbeater.cc:503] Master 127.28.55.126:46053 was elected leader, sending a full tablet report...
I20240503 08:09:49.759691 1208 raft_consensus.cc:1261] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 2 FOLLOWER]: Refusing update from remote peer 41c5a42d9b8242f3a9aa4ee972b6e850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:09:49.759970 1445 consensus_queue.cc:1035] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:09:49.771282 1283 raft_consensus.cc:1261] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 2 FOLLOWER]: Refusing update from remote peer 41c5a42d9b8242f3a9aa4ee972b6e850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:09:49.771569 1445 consensus_queue.cc:1035] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:09:50.537799 1283 consensus_queue.cc:237] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } }
I20240503 08:09:50.538425 1132 raft_consensus.cc:1261] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Refusing update from remote peer ceabd150b5d941f79238f0b7bff8a410: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:50.538486 1208 raft_consensus.cc:1261] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Refusing update from remote peer ceabd150b5d941f79238f0b7bff8a410: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:50.538664 1458 consensus_queue.cc:1035] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:09:50.538867 1459 consensus_queue.cc:1035] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:09:50.539546 1132 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 28260ffb214e412794ff375d47947ad9 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } } }
I20240503 08:09:50.539722 1208 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 28260ffb214e412794ff375d47947ad9 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } } }
I20240503 08:09:50.540747 1029 catalog_manager.cc:5484] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 reported cstate change: config changed from index -1 to 2, NON_VOTER 28260ffb214e412794ff375d47947ad9 (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "ceabd150b5d941f79238f0b7bff8a410" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } } }
W20240503 08:09:50.541854 1247 consensus_peers.cc:485] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 -> Peer 28260ffb214e412794ff375d47947ad9 (127.28.55.68:35335): Couldn't send request to peer 28260ffb214e412794ff375d47947ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c63aa1b4f3f04c50abb4c4871f64b045. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:50.560581 1437 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 28260ffb214e412794ff375d47947ad9 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } } }
I20240503 08:09:50.564908 1208 consensus_queue.cc:237] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } }
I20240503 08:09:50.565543 1283 raft_consensus.cc:1261] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Refusing update from remote peer e8406787837241b0a70b00e5943ab1dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:50.565609 1132 raft_consensus.cc:1261] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Refusing update from remote peer e8406787837241b0a70b00e5943ab1dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:50.565757 1470 consensus_queue.cc:1035] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:09:50.565979 1471 consensus_queue.cc:1035] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:09:50.566871 1283 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 28260ffb214e412794ff375d47947ad9 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } } }
I20240503 08:09:50.568078 1132 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 28260ffb214e412794ff375d47947ad9 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } } }
I20240503 08:09:50.568132 1027 catalog_manager.cc:5484] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 reported cstate change: config changed from index -1 to 2, NON_VOTER 28260ffb214e412794ff375d47947ad9 (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "e8406787837241b0a70b00e5943ab1dc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } } }
W20240503 08:09:50.568553 1172 consensus_peers.cc:485] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc -> Peer 28260ffb214e412794ff375d47947ad9 (127.28.55.68:35335): Couldn't send request to peer 28260ffb214e412794ff375d47947ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f15826aece5b461b81ae8a5629b02ebb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:09:50.616803 1470 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 28260ffb214e412794ff375d47947ad9 (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } } }
I20240503 08:09:50.617130 1475 ts_tablet_manager.cc:902] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: Initiating tablet copy from peer ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67:34171)
I20240503 08:09:50.617417 1475 tablet_copy_client.cc:308] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.67:34171
I20240503 08:09:50.620829 1293 tablet_copy_service.cc:140] P ceabd150b5d941f79238f0b7bff8a410: Received BeginTabletCopySession request for tablet c63aa1b4f3f04c50abb4c4871f64b045 from peer 28260ffb214e412794ff375d47947ad9 ({username='slave'} at 127.0.0.1:38412)
I20240503 08:09:50.620908 1293 tablet_copy_service.cc:161] P ceabd150b5d941f79238f0b7bff8a410: Beginning new tablet copy session on tablet c63aa1b4f3f04c50abb4c4871f64b045 from peer 28260ffb214e412794ff375d47947ad9 at {username='slave'} at 127.0.0.1:38412: session id = 28260ffb214e412794ff375d47947ad9-c63aa1b4f3f04c50abb4c4871f64b045
I20240503 08:09:50.621431 1293 tablet_copy_source_session.cc:205] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:50.621747 1475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63aa1b4f3f04c50abb4c4871f64b045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:50.623203 1475 tablet_copy_client.cc:783] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:50.623315 1475 tablet_copy_client.cc:647] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:50.623823 1475 tablet_copy_client.cc:523] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:50.624423 1475 tablet_bootstrap.cc:492] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: Bootstrap starting.
I20240503 08:09:50.625865 1475 tablet_bootstrap.cc:492] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:50.626026 1475 tablet_bootstrap.cc:492] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: Bootstrap complete.
I20240503 08:09:50.626130 1475 ts_tablet_manager.cc:1367] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:09:50.626291 1475 raft_consensus.cc:348] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } }
I20240503 08:09:50.626380 1475 raft_consensus.cc:729] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 28260ffb214e412794ff375d47947ad9, State: Initialized, Role: LEARNER
I20240503 08:09:50.626415 1475 consensus_queue.cc:260] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } }
I20240503 08:09:50.626591 1475 ts_tablet_manager.cc:1398] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:09:50.626796 1293 tablet_copy_service.cc:342] P ceabd150b5d941f79238f0b7bff8a410: Request end of tablet copy session 28260ffb214e412794ff375d47947ad9-c63aa1b4f3f04c50abb4c4871f64b045 received from {username='slave'} at 127.0.0.1:38412
I20240503 08:09:50.626886 1293 tablet_copy_service.cc:433] P ceabd150b5d941f79238f0b7bff8a410: ending tablet copy session 28260ffb214e412794ff375d47947ad9-c63aa1b4f3f04c50abb4c4871f64b045 on tablet c63aa1b4f3f04c50abb4c4871f64b045 with peer 28260ffb214e412794ff375d47947ad9
I20240503 08:09:50.656677 1475 ts_tablet_manager.cc:902] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: Initiating tablet copy from peer e8406787837241b0a70b00e5943ab1dc (127.28.55.66:44827)
I20240503 08:09:50.656971 1475 tablet_copy_client.cc:308] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:44827
I20240503 08:09:50.660091 1218 tablet_copy_service.cc:140] P e8406787837241b0a70b00e5943ab1dc: Received BeginTabletCopySession request for tablet f15826aece5b461b81ae8a5629b02ebb from peer 28260ffb214e412794ff375d47947ad9 ({username='slave'} at 127.0.0.1:40410)
I20240503 08:09:50.660180 1218 tablet_copy_service.cc:161] P e8406787837241b0a70b00e5943ab1dc: Beginning new tablet copy session on tablet f15826aece5b461b81ae8a5629b02ebb from peer 28260ffb214e412794ff375d47947ad9 at {username='slave'} at 127.0.0.1:40410: session id = 28260ffb214e412794ff375d47947ad9-f15826aece5b461b81ae8a5629b02ebb
I20240503 08:09:50.660688 1218 tablet_copy_source_session.cc:205] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:09:50.661001 1475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15826aece5b461b81ae8a5629b02ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:50.662359 1475 tablet_copy_client.cc:783] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: tablet copy: Starting download of 0 data blocks...
I20240503 08:09:50.662483 1475 tablet_copy_client.cc:647] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:09:50.663023 1475 tablet_copy_client.cc:523] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:09:50.663764 1475 tablet_bootstrap.cc:492] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: Bootstrap starting.
I20240503 08:09:50.665217 1475 tablet_bootstrap.cc:492] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:09:50.665396 1475 tablet_bootstrap.cc:492] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: Bootstrap complete.
I20240503 08:09:50.665499 1475 ts_tablet_manager.cc:1367] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:09:50.665670 1475 raft_consensus.cc:348] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } }
I20240503 08:09:50.665755 1475 raft_consensus.cc:729] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 28260ffb214e412794ff375d47947ad9, State: Initialized, Role: LEARNER
I20240503 08:09:50.665793 1475 consensus_queue.cc:260] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: true } }
I20240503 08:09:50.665910 1475 ts_tablet_manager.cc:1398] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:50.666069 1218 tablet_copy_service.cc:342] P e8406787837241b0a70b00e5943ab1dc: Request end of tablet copy session 28260ffb214e412794ff375d47947ad9-f15826aece5b461b81ae8a5629b02ebb received from {username='slave'} at 127.0.0.1:40410
I20240503 08:09:50.666121 1218 tablet_copy_service.cc:433] P e8406787837241b0a70b00e5943ab1dc: ending tablet copy session 28260ffb214e412794ff375d47947ad9-f15826aece5b461b81ae8a5629b02ebb on tablet f15826aece5b461b81ae8a5629b02ebb with peer 28260ffb214e412794ff375d47947ad9
I20240503 08:09:50.731145 1399 raft_consensus.cc:1203] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:09:50.760805 1399 raft_consensus.cc:1203] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:09:50.898249 1470 raft_consensus.cc:1050] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc: attempting to promote NON_VOTER 28260ffb214e412794ff375d47947ad9 to VOTER
I20240503 08:09:50.898453 1470 consensus_queue.cc:237] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } }
I20240503 08:09:50.898938 1399 raft_consensus.cc:1261] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [term 1 LEARNER]: Refusing update from remote peer e8406787837241b0a70b00e5943ab1dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:09:50.898939 1282 raft_consensus.cc:1261] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Refusing update from remote peer e8406787837241b0a70b00e5943ab1dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:09:50.899001 1132 raft_consensus.cc:1261] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Refusing update from remote peer e8406787837241b0a70b00e5943ab1dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:09:50.899183 1471 consensus_queue.cc:1035] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:09:50.899302 1481 consensus_queue.cc:1035] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:09:50.899385 1439 consensus_queue.cc:1035] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:09:50.900194 1132 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:50.900441 1399 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:50.901489 1029 catalog_manager.cc:5484] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 reported cstate change: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406787837241b0a70b00e5943ab1dc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:50.914569 1466 raft_consensus.cc:1050] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410: attempting to promote NON_VOTER 28260ffb214e412794ff375d47947ad9 to VOTER
I20240503 08:09:50.914865 1466 consensus_queue.cc:237] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } }
I20240503 08:09:50.915341 1399 raft_consensus.cc:1261] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [term 1 LEARNER]: Refusing update from remote peer ceabd150b5d941f79238f0b7bff8a410: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:09:50.915380 1132 raft_consensus.cc:1261] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Refusing update from remote peer ceabd150b5d941f79238f0b7bff8a410: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:09:50.915390 1208 raft_consensus.cc:1261] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Refusing update from remote peer ceabd150b5d941f79238f0b7bff8a410: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:09:50.915632 1466 consensus_queue.cc:1035] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:09:50.915743 1458 consensus_queue.cc:1035] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:09:50.915963 1490 consensus_queue.cc:1035] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:09:50.916687 1399 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:50.916797 1132 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:50.954654 1459 raft_consensus.cc:1013] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 1 LEADER]: attempt to promote peer 28260ffb214e412794ff375d47947ad9: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 08:09:50.978565 1282 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:50.979674 1439 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } attrs { replace: true } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:50.989732 1028 catalog_manager.cc:5484] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 reported cstate change: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ceabd150b5d941f79238f0b7bff8a410" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:50.989908 1207 consensus_queue.cc:237] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } }
I20240503 08:09:50.990453 1399 raft_consensus.cc:1261] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Refusing update from remote peer e8406787837241b0a70b00e5943ab1dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:09:50.990612 1132 raft_consensus.cc:1261] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Refusing update from remote peer e8406787837241b0a70b00e5943ab1dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:09:50.990674 1470 consensus_queue.cc:1035] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:09:50.990849 1470 consensus_queue.cc:1035] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:09:50.991434 1399 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:50.991484 1132 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:51.018783 1466 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:51.020013 1207 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 28260ffb214e412794ff375d47947ad9 (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } attrs { replace: true } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:51.026906 1029 catalog_manager.cc:5484] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 reported cstate change: config changed from index 3 to 4, VOTER ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67) evicted. New cstate: current_term: 1 leader_uuid: "e8406787837241b0a70b00e5943ab1dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:51.027351 1283 consensus_queue.cc:237] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } }
I20240503 08:09:51.027748 1399 raft_consensus.cc:1261] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Refusing update from remote peer ceabd150b5d941f79238f0b7bff8a410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:09:51.027829 1132 raft_consensus.cc:1261] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Refusing update from remote peer ceabd150b5d941f79238f0b7bff8a410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:09:51.027941 1490 consensus_queue.cc:1035] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:09:51.028057 1437 consensus_queue.cc:1035] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:09:51.028704 1132 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e8406787837241b0a70b00e5943ab1dc (127.28.55.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:51.029731 1399 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e8406787837241b0a70b00e5943ab1dc (127.28.55.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:51.097059 1271 tablet_service.cc:1514] Processing DeleteTablet for tablet f15826aece5b461b81ae8a5629b02ebb with delete_type TABLET_DATA_TOMBSTONED (TS ceabd150b5d941f79238f0b7bff8a410 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38364
I20240503 08:09:51.097152 1026 catalog_manager.cc:5484] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 reported cstate change: config changed from index 3 to 4, VOTER e8406787837241b0a70b00e5943ab1dc (127.28.55.66) evicted. New cstate: current_term: 1 leader_uuid: "ceabd150b5d941f79238f0b7bff8a410" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:51.097445 1492 tablet_replica.cc:330] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410: stopping tablet replica
I20240503 08:09:51.097540 1492 raft_consensus.cc:2229] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:51.097616 1492 raft_consensus.cc:2258] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:51.097860 1492 ts_tablet_manager.cc:1875] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:51.099678 1492 ts_tablet_manager.cc:1888] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:09:51.099745 1492 log.cc:1196] T f15826aece5b461b81ae8a5629b02ebb P ceabd150b5d941f79238f0b7bff8a410: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/f15826aece5b461b81ae8a5629b02ebb
I20240503 08:09:51.100080 1015 catalog_manager.cc:4830] TS ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67:34171): tablet f15826aece5b461b81ae8a5629b02ebb (table test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]) successfully deleted
I20240503 08:09:51.138245 1197 tablet_service.cc:1514] Processing DeleteTablet for tablet c63aa1b4f3f04c50abb4c4871f64b045 with delete_type TABLET_DATA_TOMBSTONED (TS e8406787837241b0a70b00e5943ab1dc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40378
I20240503 08:09:51.138496 1493 tablet_replica.cc:330] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc: stopping tablet replica
I20240503 08:09:51.138572 1493 raft_consensus.cc:2229] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:51.138634 1493 raft_consensus.cc:2258] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:51.138909 1493 ts_tablet_manager.cc:1875] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:51.141094 1493 ts_tablet_manager.cc:1888] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:09:51.141166 1493 log.cc:1196] T c63aa1b4f3f04c50abb4c4871f64b045 P e8406787837241b0a70b00e5943ab1dc: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/c63aa1b4f3f04c50abb4c4871f64b045
I20240503 08:09:51.141549 1015 catalog_manager.cc:4830] TS e8406787837241b0a70b00e5943ab1dc (127.28.55.66:44827): tablet c63aa1b4f3f04c50abb4c4871f64b045 (table test-workload [id=dfd70180e3bb4b82a928387473cfbcbf]) successfully deleted
I20240503 08:09:51.174197 1466 raft_consensus.cc:2932] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e8406787837241b0a70b00e5943ab1dc (127.28.55.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "ceabd150b5d941f79238f0b7bff8a410" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 34171 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:51.175230 1015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c63aa1b4f3f04c50abb4c4871f64b045 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:51.229904 1439 raft_consensus.cc:2932] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ceabd150b5d941f79238f0b7bff8a410 (127.28.55.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8406787837241b0a70b00e5943ab1dc" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44827 } } peers { permanent_uuid: "41c5a42d9b8242f3a9aa4ee972b6e850" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44877 } } peers { permanent_uuid: "28260ffb214e412794ff375d47947ad9" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 35335 } attrs { promote: false } } }
I20240503 08:09:51.231010 1015 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f15826aece5b461b81ae8a5629b02ebb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:09:51.538110 1083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:51.538295 1083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:51.538326 1083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:53.538501 1083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:53.538669 1083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:53.538702 1083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:55.538897 1083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:55.539072 1083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:55.539105 1083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:57.539269 1083 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:57.539419 1083 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:57.539444 1083 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:09:57.800072 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:09:57.803364 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:57.803579 28893 tablet_replica.cc:330] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850: stopping tablet replica
I20240503 08:09:57.803671 28893 raft_consensus.cc:2229] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:57.803748 28893 raft_consensus.cc:2258] T f15826aece5b461b81ae8a5629b02ebb P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.803941 28893 tablet_replica.cc:330] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850: stopping tablet replica
I20240503 08:09:57.803973 28893 raft_consensus.cc:2229] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:57.804013 28893 raft_consensus.cc:2258] T c63aa1b4f3f04c50abb4c4871f64b045 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.804111 28893 tablet_replica.cc:330] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850: stopping tablet replica
I20240503 08:09:57.804141 28893 raft_consensus.cc:2229] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:09:57.804203 28893 raft_consensus.cc:2258] T 3e756374f39045bf8e444601b2b53d72 P 41c5a42d9b8242f3a9aa4ee972b6e850 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.816440 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:09:57.817724 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:09:57.821019 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:57.821159 28893 tablet_replica.cc:330] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc: stopping tablet replica
I20240503 08:09:57.821218 28893 raft_consensus.cc:2229] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:57.821293 28893 raft_consensus.cc:2258] T f15826aece5b461b81ae8a5629b02ebb P e8406787837241b0a70b00e5943ab1dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.821480 28893 tablet_replica.cc:330] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc: stopping tablet replica
I20240503 08:09:57.821512 28893 raft_consensus.cc:2229] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:57.821543 28893 raft_consensus.cc:2258] T 3e756374f39045bf8e444601b2b53d72 P e8406787837241b0a70b00e5943ab1dc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.833600 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:09:57.835036 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:09:57.838733 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:57.838904 28893 tablet_replica.cc:330] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410: stopping tablet replica
I20240503 08:09:57.838960 28893 raft_consensus.cc:2229] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:57.839025 28893 raft_consensus.cc:2258] T 3e756374f39045bf8e444601b2b53d72 P ceabd150b5d941f79238f0b7bff8a410 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.839222 28893 tablet_replica.cc:330] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410: stopping tablet replica
I20240503 08:09:57.839272 28893 raft_consensus.cc:2229] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:57.839340 28893 raft_consensus.cc:2258] T c63aa1b4f3f04c50abb4c4871f64b045 P ceabd150b5d941f79238f0b7bff8a410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.851363 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:09:57.852517 28893 tablet_server.cc:178] TabletServer@127.28.55.68:0 shutting down...
I20240503 08:09:57.855741 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:57.855901 28893 tablet_replica.cc:330] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9: stopping tablet replica
I20240503 08:09:57.855981 28893 raft_consensus.cc:2229] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:57.856038 28893 raft_consensus.cc:2258] T f15826aece5b461b81ae8a5629b02ebb P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.856220 28893 tablet_replica.cc:330] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9: stopping tablet replica
I20240503 08:09:57.856249 28893 raft_consensus.cc:2229] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:57.856276 28893 raft_consensus.cc:2258] T c63aa1b4f3f04c50abb4c4871f64b045 P 28260ffb214e412794ff375d47947ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.868336 28893 tablet_server.cc:194] TabletServer@127.28.55.68:0 shutdown complete.
I20240503 08:09:57.869525 28893 master.cc:537] Master@127.28.55.126:46053 shutting down...
I20240503 08:09:57.871994 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:57.872072 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:57.872092 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3320b646552142ea8f51cedce3652e5d: stopping tablet replica
I20240503 08:09:57.883832 28893 master.cc:552] Master@127.28.55.126:46053 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10388 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20240503 08:09:57.889737 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:42017
I20240503 08:09:57.889906 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:57.891422 1498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:57.891464 1496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:57.891506 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:57.891484 1495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:57.891664 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:57.891701 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:57.891752 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723797891753 us; error 0 us; skew 500 ppm
I20240503 08:09:57.896088 28893 webserver.cc:460] Webserver started at http://127.28.55.126:36773/ using document root <none> and password file <none>
I20240503 08:09:57.896175 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:57.896219 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:57.896265 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:57.896543 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "d15696474c9349c7acdb2e71960b3e88"
format_stamp: "Formatted at 2024-05-03 08:09:57 on dist-test-slave-mlgg"
I20240503 08:09:57.897455 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:57.897951 1503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.898104 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.898146 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "d15696474c9349c7acdb2e71960b3e88"
format_stamp: "Formatted at 2024-05-03 08:09:57 on dist-test-slave-mlgg"
I20240503 08:09:57.898192 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:57.917197 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:57.917433 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:57.920102 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:42017
I20240503 08:09:57.923074 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:42017 every 8 connection(s)
I20240503 08:09:57.923223 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:57.924294 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88: Bootstrap starting.
I20240503 08:09:57.924561 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:57.925042 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88: No bootstrap required, opened a new log
I20240503 08:09:57.925158 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15696474c9349c7acdb2e71960b3e88" member_type: VOTER }
I20240503 08:09:57.925211 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:57.925223 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15696474c9349c7acdb2e71960b3e88, State: Initialized, Role: FOLLOWER
I20240503 08:09:57.925257 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15696474c9349c7acdb2e71960b3e88" member_type: VOTER }
I20240503 08:09:57.925290 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:09:57.925307 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:09:57.925323 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:57.925797 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15696474c9349c7acdb2e71960b3e88" member_type: VOTER }
I20240503 08:09:57.925853 1556 leader_election.cc:302] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d15696474c9349c7acdb2e71960b3e88; no voters:
I20240503 08:09:57.925953 1556 leader_election.cc:288] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:09:57.926005 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:57.926106 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:09:57.926142 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 1 LEADER]: Becoming Leader. State: Replica: d15696474c9349c7acdb2e71960b3e88, State: Running, Role: LEADER
I20240503 08:09:57.926190 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15696474c9349c7acdb2e71960b3e88" member_type: VOTER }
I20240503 08:09:57.926403 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d15696474c9349c7acdb2e71960b3e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15696474c9349c7acdb2e71960b3e88" member_type: VOTER } }
I20240503 08:09:57.926429 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d15696474c9349c7acdb2e71960b3e88. Latest consensus state: current_term: 1 leader_uuid: "d15696474c9349c7acdb2e71960b3e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15696474c9349c7acdb2e71960b3e88" member_type: VOTER } }
I20240503 08:09:57.926481 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:57.926489 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:57.926734 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:09:57.926908 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:09:57.927446 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:09:57.927470 1564 catalog_manager.cc:1347] Generated new cluster ID: d21cd85614a04c88ad10f71a8394ec3c
I20240503 08:09:57.927510 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:09:57.940812 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:09:57.941335 1564 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:09:57.949676 1564 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88: Generated new TSK 0
I20240503 08:09:57.949791 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:09:57.959450 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:57.960906 1577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:57.960922 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:57.961071 1578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:57.961099 1580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:57.961230 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:57.961270 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:57.961282 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723797961282 us; error 0 us; skew 500 ppm
I20240503 08:09:57.961990 28893 webserver.cc:460] Webserver started at http://127.28.55.65:45549/ using document root <none> and password file <none>
I20240503 08:09:57.962064 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:57.962113 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:57.962162 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:57.962450 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "f2ede68872194c2e8ce8043e9cfbef2f"
format_stamp: "Formatted at 2024-05-03 08:09:57 on dist-test-slave-mlgg"
I20240503 08:09:57.963330 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:57.963728 1585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.963863 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.963902 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "f2ede68872194c2e8ce8043e9cfbef2f"
format_stamp: "Formatted at 2024-05-03 08:09:57 on dist-test-slave-mlgg"
I20240503 08:09:57.963950 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:57.970055 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:57.970268 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:57.970417 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:57.970618 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:57.970654 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.970681 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:57.970706 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.973381 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:44107
I20240503 08:09:57.973438 1648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:44107 every 8 connection(s)
I20240503 08:09:57.973774 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:57.975076 1654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:57.975090 1653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:57.975605 28893 server_base.cc:1034] running on GCE node
W20240503 08:09:57.975883 1656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:57.976055 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:57.976109 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:57.976136 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723797976137 us; error 0 us; skew 500 ppm
I20240503 08:09:57.976917 28893 webserver.cc:460] Webserver started at http://127.28.55.66:33453/ using document root <none> and password file <none>
I20240503 08:09:57.977007 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:57.977056 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:57.977103 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:57.977389 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "e6f3f8a532d0464aa454b7c5e9935a5b"
format_stamp: "Formatted at 2024-05-03 08:09:57 on dist-test-slave-mlgg"
I20240503 08:09:57.977452 1649 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42017
I20240503 08:09:57.977510 1649 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:57.977624 1649 heartbeater.cc:511] Master 127.28.55.126:42017 requested a full tablet report, sending...
I20240503 08:09:57.977843 1520 ts_manager.cc:194] Registered new tserver with Master: f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107)
I20240503 08:09:57.978346 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:57.978500 1520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41638
I20240503 08:09:57.978921 1661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.979043 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:09:57.979084 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "e6f3f8a532d0464aa454b7c5e9935a5b"
format_stamp: "Formatted at 2024-05-03 08:09:57 on dist-test-slave-mlgg"
I20240503 08:09:57.979139 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:57.990883 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:57.991113 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:57.991271 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:57.991474 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:57.991500 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.991520 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:57.991533 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.994146 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:46085
I20240503 08:09:57.994206 1724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:46085 every 8 connection(s)
I20240503 08:09:57.994513 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:57.995640 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:57.995756 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:57.995813 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:57.996035 28893 server_base.cc:1034] running on GCE node
I20240503 08:09:57.996191 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:57.996253 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:57.996280 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723797996280 us; error 0 us; skew 500 ppm
I20240503 08:09:57.997064 28893 webserver.cc:460] Webserver started at http://127.28.55.67:44353/ using document root <none> and password file <none>
I20240503 08:09:57.997218 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:57.997295 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:57.997383 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:57.997727 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "94872d9dc94f4af58445ad70b17ca91e"
format_stamp: "Formatted at 2024-05-03 08:09:57 on dist-test-slave-mlgg"
I20240503 08:09:57.997730 1725 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42017
I20240503 08:09:57.997843 1725 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:57.997965 1725 heartbeater.cc:511] Master 127.28.55.126:42017 requested a full tablet report, sending...
I20240503 08:09:57.998176 1520 ts_manager.cc:194] Registered new tserver with Master: e6f3f8a532d0464aa454b7c5e9935a5b (127.28.55.66:46085)
I20240503 08:09:57.998529 1520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41654
I20240503 08:09:57.998718 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:57.999217 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.999336 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:57.999377 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "94872d9dc94f4af58445ad70b17ca91e"
format_stamp: "Formatted at 2024-05-03 08:09:57 on dist-test-slave-mlgg"
I20240503 08:09:57.999420 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:58.016402 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:58.016667 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:58.016827 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:58.017024 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:58.017050 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:58.017079 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:58.017103 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:58.020238 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:37531
I20240503 08:09:58.020321 1799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:37531 every 8 connection(s)
I20240503 08:09:58.023416 1800 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42017
I20240503 08:09:58.023490 1800 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:58.023600 1800 heartbeater.cc:511] Master 127.28.55.126:42017 requested a full tablet report, sending...
I20240503 08:09:58.023844 1520 ts_manager.cc:194] Registered new tserver with Master: 94872d9dc94f4af58445ad70b17ca91e (127.28.55.67:37531)
I20240503 08:09:58.024191 1520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41666
I20240503 08:09:58.024899 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004477765s
I20240503 08:09:58.979178 1649 heartbeater.cc:503] Master 127.28.55.126:42017 was elected leader, sending a full tablet report...
I20240503 08:09:58.999078 1725 heartbeater.cc:503] Master 127.28.55.126:42017 was elected leader, sending a full tablet report...
I20240503 08:09:59.024719 1800 heartbeater.cc:503] Master 127.28.55.126:42017 was elected leader, sending a full tablet report...
I20240503 08:09:59.050283 28893 test_util.cc:268] Using random seed: 1055795233
I20240503 08:09:59.055534 1520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41682:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:09:59.055693 1520 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:09:59.060601 1689 tablet_service.cc:1467] Processing CreateTablet for tablet bb6b1e57d05f4e0e89ffd4db0e3eee20 (DEFAULT_TABLE table=test-workload [id=91dee709bae64b9c834e548a6d114aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:59.060776 1689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6b1e57d05f4e0e89ffd4db0e3eee20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:59.060797 1613 tablet_service.cc:1467] Processing CreateTablet for tablet bb6b1e57d05f4e0e89ffd4db0e3eee20 (DEFAULT_TABLE table=test-workload [id=91dee709bae64b9c834e548a6d114aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:59.060873 1613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6b1e57d05f4e0e89ffd4db0e3eee20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:59.062337 1764 tablet_service.cc:1467] Processing CreateTablet for tablet bb6b1e57d05f4e0e89ffd4db0e3eee20 (DEFAULT_TABLE table=test-workload [id=91dee709bae64b9c834e548a6d114aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:59.062386 1821 tablet_bootstrap.cc:492] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f: Bootstrap starting.
I20240503 08:09:59.062458 1764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6b1e57d05f4e0e89ffd4db0e3eee20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:59.062788 1821 tablet_bootstrap.cc:654] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:59.062345 1820 tablet_bootstrap.cc:492] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b: Bootstrap starting.
I20240503 08:09:59.063366 1820 tablet_bootstrap.cc:654] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:59.063616 1821 tablet_bootstrap.cc:492] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f: No bootstrap required, opened a new log
I20240503 08:09:59.063669 1821 ts_tablet_manager.cc:1367] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:59.063745 1824 tablet_bootstrap.cc:492] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e: Bootstrap starting.
I20240503 08:09:59.063951 1820 tablet_bootstrap.cc:492] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b: No bootstrap required, opened a new log
I20240503 08:09:59.063915 1821 raft_consensus.cc:348] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.064000 1820 ts_tablet_manager.cc:1367] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:59.064002 1821 raft_consensus.cc:374] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:59.064077 1821 raft_consensus.cc:729] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ede68872194c2e8ce8043e9cfbef2f, State: Initialized, Role: FOLLOWER
I20240503 08:09:59.064126 1821 consensus_queue.cc:260] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.064168 1820 raft_consensus.cc:348] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.064211 1820 raft_consensus.cc:374] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:59.064229 1820 raft_consensus.cc:729] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f3f8a532d0464aa454b7c5e9935a5b, State: Initialized, Role: FOLLOWER
I20240503 08:09:59.064270 1820 consensus_queue.cc:260] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.064288 1821 ts_tablet_manager.cc:1398] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:59.064411 1820 ts_tablet_manager.cc:1398] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:59.064839 1824 tablet_bootstrap.cc:654] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:59.065349 1824 tablet_bootstrap.cc:492] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e: No bootstrap required, opened a new log
I20240503 08:09:59.065387 1824 ts_tablet_manager.cc:1367] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:59.065493 1824 raft_consensus.cc:348] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.065536 1824 raft_consensus.cc:374] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:59.065546 1824 raft_consensus.cc:729] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94872d9dc94f4af58445ad70b17ca91e, State: Initialized, Role: FOLLOWER
I20240503 08:09:59.065599 1824 consensus_queue.cc:260] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.065774 1824 ts_tablet_manager.cc:1398] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:59.425171 1825 raft_consensus.cc:482] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:59.425308 1825 raft_consensus.cc:504] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.425546 1825 leader_election.cc:288] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94872d9dc94f4af58445ad70b17ca91e (127.28.55.67:37531), e6f3f8a532d0464aa454b7c5e9935a5b (127.28.55.66:46085)
I20240503 08:09:59.428566 1828 raft_consensus.cc:482] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:59.428593 1774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6b1e57d05f4e0e89ffd4db0e3eee20" candidate_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94872d9dc94f4af58445ad70b17ca91e" is_pre_election: true
I20240503 08:09:59.428566 1699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6b1e57d05f4e0e89ffd4db0e3eee20" candidate_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" is_pre_election: true
I20240503 08:09:59.428691 1828 raft_consensus.cc:504] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.428763 1699 raft_consensus.cc:2454] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ede68872194c2e8ce8043e9cfbef2f in term 0.
I20240503 08:09:59.428792 1774 raft_consensus.cc:2454] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ede68872194c2e8ce8043e9cfbef2f in term 0.
I20240503 08:09:59.428879 1828 leader_election.cc:288] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6f3f8a532d0464aa454b7c5e9935a5b (127.28.55.66:46085), f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107)
I20240503 08:09:59.428920 1587 leader_election.cc:302] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6f3f8a532d0464aa454b7c5e9935a5b, f2ede68872194c2e8ce8043e9cfbef2f; no voters:
I20240503 08:09:59.429025 1825 raft_consensus.cc:2781] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:59.429061 1825 raft_consensus.cc:482] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:59.429080 1825 raft_consensus.cc:3037] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:59.429993 1825 raft_consensus.cc:504] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.430133 1825 leader_election.cc:288] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [CANDIDATE]: Term 1 election: Requested vote from peers 94872d9dc94f4af58445ad70b17ca91e (127.28.55.67:37531), e6f3f8a532d0464aa454b7c5e9935a5b (127.28.55.66:46085)
I20240503 08:09:59.430316 1774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6b1e57d05f4e0e89ffd4db0e3eee20" candidate_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94872d9dc94f4af58445ad70b17ca91e"
I20240503 08:09:59.430375 1774 raft_consensus.cc:3037] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:59.430557 1699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6b1e57d05f4e0e89ffd4db0e3eee20" candidate_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b"
I20240503 08:09:59.430606 1699 raft_consensus.cc:3037] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:59.431197 1774 raft_consensus.cc:2454] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ede68872194c2e8ce8043e9cfbef2f in term 1.
I20240503 08:09:59.431350 1699 raft_consensus.cc:2454] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ede68872194c2e8ce8043e9cfbef2f in term 1.
I20240503 08:09:59.431491 1587 leader_election.cc:302] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6f3f8a532d0464aa454b7c5e9935a5b, f2ede68872194c2e8ce8043e9cfbef2f; no voters:
I20240503 08:09:59.431583 1825 raft_consensus.cc:2781] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:59.431722 1825 raft_consensus.cc:686] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 1 LEADER]: Becoming Leader. State: Replica: f2ede68872194c2e8ce8043e9cfbef2f, State: Running, Role: LEADER
I20240503 08:09:59.431782 1825 consensus_queue.cc:237] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:09:59.432404 1520 catalog_manager.cc:5484] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f reported cstate change: term changed from 0 to 1, leader changed from <none> to f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65). New cstate: current_term: 1 leader_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:59.432528 1623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6b1e57d05f4e0e89ffd4db0e3eee20" candidate_uuid: "94872d9dc94f4af58445ad70b17ca91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" is_pre_election: true
I20240503 08:09:59.432435 1699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6b1e57d05f4e0e89ffd4db0e3eee20" candidate_uuid: "94872d9dc94f4af58445ad70b17ca91e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" is_pre_election: true
I20240503 08:09:59.432654 1699 raft_consensus.cc:2379] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94872d9dc94f4af58445ad70b17ca91e in current term 1: Already voted for candidate f2ede68872194c2e8ce8043e9cfbef2f in this term.
I20240503 08:09:59.432790 1738 leader_election.cc:302] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94872d9dc94f4af58445ad70b17ca91e; no voters: e6f3f8a532d0464aa454b7c5e9935a5b, f2ede68872194c2e8ce8043e9cfbef2f
I20240503 08:09:59.433001 1828 raft_consensus.cc:2726] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:09:59.497709 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:09:59.500304 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:59.500463 28893 tablet_replica.cc:330] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f: stopping tablet replica
I20240503 08:09:59.500533 28893 raft_consensus.cc:2229] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:59.500592 28893 pending_rounds.cc:62] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f: Trying to abort 1 pending ops.
I20240503 08:09:59.500603 28893 pending_rounds.cc:68] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508682473033728 op_type: NO_OP noop_request { }
I20240503 08:09:59.500649 28893 raft_consensus.cc:2866] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:09:59.500664 28893 raft_consensus.cc:2258] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P f2ede68872194c2e8ce8043e9cfbef2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:59.514863 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:09:59.927501 1574 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:09:59.927668 1574 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:09:59.927708 1574 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:10:00.933598 1835 raft_consensus.cc:482] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:00.933709 1835 raft_consensus.cc:504] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:10:00.933935 1835 leader_election.cc:288] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94872d9dc94f4af58445ad70b17ca91e (127.28.55.67:37531), f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107)
W20240503 08:10:00.934315 1662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.65:44107: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 08:10:00.934783 1662 leader_election.cc:334] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107): Network error: Client connection negotiation failed: client connection to 127.28.55.65:44107: connect: Connection refused (error 111)
I20240503 08:10:00.936728 1774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6b1e57d05f4e0e89ffd4db0e3eee20" candidate_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94872d9dc94f4af58445ad70b17ca91e" is_pre_election: true
I20240503 08:10:00.936816 1774 raft_consensus.cc:2454] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6f3f8a532d0464aa454b7c5e9935a5b in term 1.
I20240503 08:10:00.936952 1665 leader_election.cc:302] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 94872d9dc94f4af58445ad70b17ca91e, e6f3f8a532d0464aa454b7c5e9935a5b; no voters: f2ede68872194c2e8ce8043e9cfbef2f
I20240503 08:10:00.937065 1835 raft_consensus.cc:2781] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:10:00.937103 1835 raft_consensus.cc:482] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:00.937116 1835 raft_consensus.cc:3037] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:00.937951 1835 raft_consensus.cc:504] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:10:00.938110 1835 leader_election.cc:288] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [CANDIDATE]: Term 2 election: Requested vote from peers 94872d9dc94f4af58445ad70b17ca91e (127.28.55.67:37531), f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107)
I20240503 08:10:00.938236 1774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6b1e57d05f4e0e89ffd4db0e3eee20" candidate_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94872d9dc94f4af58445ad70b17ca91e"
I20240503 08:10:00.938306 1774 raft_consensus.cc:3037] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:10:00.938542 1662 leader_election.cc:334] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107): Network error: Client connection negotiation failed: client connection to 127.28.55.65:44107: connect: Connection refused (error 111)
I20240503 08:10:00.938956 1774 raft_consensus.cc:2454] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6f3f8a532d0464aa454b7c5e9935a5b in term 2.
I20240503 08:10:00.939110 1665 leader_election.cc:302] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 94872d9dc94f4af58445ad70b17ca91e, e6f3f8a532d0464aa454b7c5e9935a5b; no voters: f2ede68872194c2e8ce8043e9cfbef2f
I20240503 08:10:00.939193 1835 raft_consensus.cc:2781] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:00.939321 1835 raft_consensus.cc:686] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 2 LEADER]: Becoming Leader. State: Replica: e6f3f8a532d0464aa454b7c5e9935a5b, State: Running, Role: LEADER
I20240503 08:10:00.939365 1835 consensus_queue.cc:237] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } }
I20240503 08:10:00.939913 1520 catalog_manager.cc:5484] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b reported cstate change: term changed from 1 to 2, leader changed from f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65) to e6f3f8a532d0464aa454b7c5e9935a5b (127.28.55.66). New cstate: current_term: 2 leader_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6f3f8a532d0464aa454b7c5e9935a5b" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 46085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2ede68872194c2e8ce8043e9cfbef2f" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 44107 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:01.508065 1774 raft_consensus.cc:1261] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 2 FOLLOWER]: Refusing update from remote peer e6f3f8a532d0464aa454b7c5e9935a5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 08:10:01.508440 1846 consensus_queue.cc:1035] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "94872d9dc94f4af58445ad70b17ca91e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 37531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:01.510562 1662 consensus_peers.cc:485] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b -> Peer f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107): Couldn't send request to peer f2ede68872194c2e8ce8043e9cfbef2f. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:44107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:01.927899 1574 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:10:01.928056 1574 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:10:01.928092 1574 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:10:03.928283 1574 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:10:03.928432 1574 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:10:03.928462 1574 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 08:10:04.085135 1662 consensus_peers.cc:485] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b -> Peer f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107): Couldn't send request to peer f2ede68872194c2e8ce8043e9cfbef2f. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:44107: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:10:05.928961 1574 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:10:05.929103 1574 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:10:05.929126 1574 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 08:10:06.131412 1662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.65:44107: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:10:06.521473 1662 consensus_peers.cc:485] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b -> Peer f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107): Couldn't send request to peer f2ede68872194c2e8ce8043e9cfbef2f. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:44107: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:10:07.929273 1574 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:10:07.929419 1574 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:10:07.929442 1574 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 08:10:08.826334 1662 consensus_peers.cc:485] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b -> Peer f2ede68872194c2e8ce8043e9cfbef2f (127.28.55.65:44107): Couldn't send request to peer f2ede68872194c2e8ce8043e9cfbef2f. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:44107: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 08:10:09.929643 1574 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 08:10:09.929826 1574 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 08:10:09.929867 1574 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 08:10:10.567472 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:10:10.570075 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:10.570281 28893 tablet_replica.cc:330] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b: stopping tablet replica
I20240503 08:10:10.570363 28893 raft_consensus.cc:2229] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:10:10.570438 28893 raft_consensus.cc:2258] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P e6f3f8a532d0464aa454b7c5e9935a5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:10.582413 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:10:10.583703 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:10:10.586455 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:10.586606 28893 tablet_replica.cc:330] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e: stopping tablet replica
I20240503 08:10:10.586673 28893 raft_consensus.cc:2229] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:10.586727 28893 raft_consensus.cc:2258] T bb6b1e57d05f4e0e89ffd4db0e3eee20 P 94872d9dc94f4af58445ad70b17ca91e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:10.598477 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:10:10.599555 28893 master.cc:537] Master@127.28.55.126:42017 shutting down...
I20240503 08:10:10.601640 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:10.601728 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:10.601761 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P d15696474c9349c7acdb2e71960b3e88: stopping tablet replica
I20240503 08:10:10.613338 28893 master.cc:552] Master@127.28.55.126:42017 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (12728 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20240503 08:10:10.617808 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:45797
I20240503 08:10:10.617980 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:10.619103 1873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:10.619240 1870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:10.619400 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:10.619266 1871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:10.619578 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:10.619616 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:10.619632 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723810619632 us; error 0 us; skew 500 ppm
I20240503 08:10:10.620306 28893 webserver.cc:460] Webserver started at http://127.28.55.126:46863/ using document root <none> and password file <none>
I20240503 08:10:10.620383 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:10.620430 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:10.620497 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:10.620805 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "a9862686ac8d4bb6883555c1bd6a77c6"
format_stamp: "Formatted at 2024-05-03 08:10:10 on dist-test-slave-mlgg"
I20240503 08:10:10.621665 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:10.622206 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.622328 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.622372 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "a9862686ac8d4bb6883555c1bd6a77c6"
format_stamp: "Formatted at 2024-05-03 08:10:10 on dist-test-slave-mlgg"
I20240503 08:10:10.622416 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:10.634925 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:10.635131 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:10.637679 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:45797
I20240503 08:10:10.639429 1930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:45797 every 8 connection(s)
I20240503 08:10:10.639698 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:10.640774 1931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6: Bootstrap starting.
I20240503 08:10:10.641034 1931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:10.641589 1931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6: No bootstrap required, opened a new log
I20240503 08:10:10.641707 1931 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9862686ac8d4bb6883555c1bd6a77c6" member_type: VOTER }
I20240503 08:10:10.641760 1931 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:10.641772 1931 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9862686ac8d4bb6883555c1bd6a77c6, State: Initialized, Role: FOLLOWER
I20240503 08:10:10.641811 1931 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9862686ac8d4bb6883555c1bd6a77c6" member_type: VOTER }
I20240503 08:10:10.641844 1931 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:10:10.641860 1931 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:10:10.641884 1931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:10.642381 1931 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9862686ac8d4bb6883555c1bd6a77c6" member_type: VOTER }
I20240503 08:10:10.642436 1931 leader_election.cc:302] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9862686ac8d4bb6883555c1bd6a77c6; no voters:
I20240503 08:10:10.642524 1931 leader_election.cc:288] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:10:10.642592 1934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:10.642652 1931 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:10:10.642736 1934 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 1 LEADER]: Becoming Leader. State: Replica: a9862686ac8d4bb6883555c1bd6a77c6, State: Running, Role: LEADER
I20240503 08:10:10.642792 1934 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9862686ac8d4bb6883555c1bd6a77c6" member_type: VOTER }
I20240503 08:10:10.643050 1935 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9862686ac8d4bb6883555c1bd6a77c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9862686ac8d4bb6883555c1bd6a77c6" member_type: VOTER } }
I20240503 08:10:10.643074 1936 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9862686ac8d4bb6883555c1bd6a77c6. Latest consensus state: current_term: 1 leader_uuid: "a9862686ac8d4bb6883555c1bd6a77c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9862686ac8d4bb6883555c1bd6a77c6" member_type: VOTER } }
I20240503 08:10:10.643117 1935 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:10.643127 1936 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:10.643399 1940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:10:10.643570 1940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:10:10.643990 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:10:10.644114 1940 catalog_manager.cc:1347] Generated new cluster ID: 897feb634e3848c9baf2b20b2f5c1181
I20240503 08:10:10.644160 1940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:10:10.648785 1940 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:10:10.649192 1940 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:10:10.653779 1940 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6: Generated new TSK 0
I20240503 08:10:10.653882 1940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:10:10.659792 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:10.661381 1955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:10.661509 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:10.661551 1953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:10.661494 1952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:10.661782 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:10.661820 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:10.661837 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723810661837 us; error 0 us; skew 500 ppm
I20240503 08:10:10.662484 28893 webserver.cc:460] Webserver started at http://127.28.55.65:44459/ using document root <none> and password file <none>
I20240503 08:10:10.662571 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:10.662616 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:10.662667 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:10.663004 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "685c5510f4594a3e96c849a3c203118b"
format_stamp: "Formatted at 2024-05-03 08:10:10 on dist-test-slave-mlgg"
I20240503 08:10:10.663821 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:10.664413 1960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.664562 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:10.664608 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "685c5510f4594a3e96c849a3c203118b"
format_stamp: "Formatted at 2024-05-03 08:10:10 on dist-test-slave-mlgg"
I20240503 08:10:10.664655 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:10.669117 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:10.669307 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:10.669448 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:10.669641 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:10.669667 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.669692 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:10.669718 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.672299 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:46523
I20240503 08:10:10.672353 2023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:46523 every 8 connection(s)
I20240503 08:10:10.672663 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:10.673889 2031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:10.674005 2029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:10.674016 2028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:10.674192 28893 server_base.cc:1034] running on GCE node
I20240503 08:10:10.674387 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:10.674418 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:10.674436 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723810674436 us; error 0 us; skew 500 ppm
I20240503 08:10:10.675199 28893 webserver.cc:460] Webserver started at http://127.28.55.66:37527/ using document root <none> and password file <none>
I20240503 08:10:10.675271 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:10.675355 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:10.675406 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:10.675670 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "f3acedb5ad844e60b5f7a0b72383a6b3"
format_stamp: "Formatted at 2024-05-03 08:10:10 on dist-test-slave-mlgg"
I20240503 08:10:10.676330 2024 heartbeater.cc:347] Connected to a master server at 127.28.55.126:45797
I20240503 08:10:10.676396 2024 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:10.676497 2024 heartbeater.cc:511] Master 127.28.55.126:45797 requested a full tablet report, sending...
I20240503 08:10:10.676704 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:10.676740 1895 ts_manager.cc:194] Registered new tserver with Master: 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523)
I20240503 08:10:10.677220 2036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.677341 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.677387 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "f3acedb5ad844e60b5f7a0b72383a6b3"
format_stamp: "Formatted at 2024-05-03 08:10:10 on dist-test-slave-mlgg"
I20240503 08:10:10.677430 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:10.677505 1895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56958
I20240503 08:10:10.688103 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:10.688294 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:10.688483 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:10.688650 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:10.688678 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.688696 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:10.688712 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.691469 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:37521
I20240503 08:10:10.691524 2099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:37521 every 8 connection(s)
I20240503 08:10:10.691812 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:10.693234 2103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:10.693220 2104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:10.693482 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:10.693189 2106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:10.694245 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:10.694288 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:10.694305 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723810694306 us; error 0 us; skew 500 ppm
I20240503 08:10:10.694921 28893 webserver.cc:460] Webserver started at http://127.28.55.67:43455/ using document root <none> and password file <none>
I20240503 08:10:10.695012 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:10.695058 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:10.695140 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:10.695184 2100 heartbeater.cc:347] Connected to a master server at 127.28.55.126:45797
I20240503 08:10:10.695248 2100 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:10.695459 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "3fe2aae583d14ad2a6892ae954d49708"
format_stamp: "Formatted at 2024-05-03 08:10:10 on dist-test-slave-mlgg"
I20240503 08:10:10.695461 2100 heartbeater.cc:511] Master 127.28.55.126:45797 requested a full tablet report, sending...
I20240503 08:10:10.695781 1895 ts_manager.cc:194] Registered new tserver with Master: f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521)
I20240503 08:10:10.696131 1895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56968
I20240503 08:10:10.696434 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:10.696880 2111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.697001 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.697043 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "3fe2aae583d14ad2a6892ae954d49708"
format_stamp: "Formatted at 2024-05-03 08:10:10 on dist-test-slave-mlgg"
I20240503 08:10:10.697089 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:10.706983 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:10.707229 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:10.707422 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:10.707669 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:10.707696 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.707724 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:10.707748 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:10.710912 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:45635
I20240503 08:10:10.710961 2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:45635 every 8 connection(s)
I20240503 08:10:10.714066 2175 heartbeater.cc:347] Connected to a master server at 127.28.55.126:45797
I20240503 08:10:10.714131 2175 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:10.714241 2175 heartbeater.cc:511] Master 127.28.55.126:45797 requested a full tablet report, sending...
I20240503 08:10:10.714540 1895 ts_manager.cc:194] Registered new tserver with Master: 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:10.714923 1895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56982
I20240503 08:10:10.715483 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004378069s
I20240503 08:10:11.678061 2024 heartbeater.cc:503] Master 127.28.55.126:45797 was elected leader, sending a full tablet report...
I20240503 08:10:11.696650 2100 heartbeater.cc:503] Master 127.28.55.126:45797 was elected leader, sending a full tablet report...
I20240503 08:10:11.715453 2175 heartbeater.cc:503] Master 127.28.55.126:45797 was elected leader, sending a full tablet report...
I20240503 08:10:11.740481 28893 test_util.cc:268] Using random seed: 1068485436
I20240503 08:10:11.746197 1895 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56996:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:10:11.746383 1895 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:10:11.751648 1988 tablet_service.cc:1467] Processing CreateTablet for tablet 5a662efda5fc4f92a84284a3ffb1ef4b (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:11.751713 1985 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd1d3f3edc54adbae4a580531a7b21b (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:11.751874 1988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a662efda5fc4f92a84284a3ffb1ef4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.752753 2064 tablet_service.cc:1467] Processing CreateTablet for tablet 5a662efda5fc4f92a84284a3ffb1ef4b (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:11.752804 2062 tablet_service.cc:1467] Processing CreateTablet for tablet 4416cf3a68f546e49bf05e9e74b08773 (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:11.752758 2063 tablet_service.cc:1467] Processing CreateTablet for tablet 8cafb41bf92a4f36877bf48362aee7a1 (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:11.752898 2064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a662efda5fc4f92a84284a3ffb1ef4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.752938 2063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cafb41bf92a4f36877bf48362aee7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.752983 2061 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd1d3f3edc54adbae4a580531a7b21b (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:11.752995 2062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4416cf3a68f546e49bf05e9e74b08773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.753130 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd1d3f3edc54adbae4a580531a7b21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.751679 1986 tablet_service.cc:1467] Processing CreateTablet for tablet 4416cf3a68f546e49bf05e9e74b08773 (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:11.751648 1987 tablet_service.cc:1467] Processing CreateTablet for tablet 8cafb41bf92a4f36877bf48362aee7a1 (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:11.754096 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4416cf3a68f546e49bf05e9e74b08773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.755023 2139 tablet_service.cc:1467] Processing CreateTablet for tablet 5a662efda5fc4f92a84284a3ffb1ef4b (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:11.755049 1987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cafb41bf92a4f36877bf48362aee7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.755118 2139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a662efda5fc4f92a84284a3ffb1ef4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.755437 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd1d3f3edc54adbae4a580531a7b21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.756173 2196 tablet_bootstrap.cc:492] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3: Bootstrap starting.
I20240503 08:10:11.756497 2138 tablet_service.cc:1467] Processing CreateTablet for tablet 8cafb41bf92a4f36877bf48362aee7a1 (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:11.756551 2196 tablet_bootstrap.cc:654] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.756599 2138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cafb41bf92a4f36877bf48362aee7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.756814 2197 tablet_bootstrap.cc:492] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708: Bootstrap starting.
I20240503 08:10:11.756875 2195 tablet_bootstrap.cc:492] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b: Bootstrap starting.
I20240503 08:10:11.757207 2195 tablet_bootstrap.cc:654] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.757212 2197 tablet_bootstrap.cc:654] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.757458 2137 tablet_service.cc:1467] Processing CreateTablet for tablet 4416cf3a68f546e49bf05e9e74b08773 (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:11.757548 2137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4416cf3a68f546e49bf05e9e74b08773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.758177 2196 tablet_bootstrap.cc:492] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3: No bootstrap required, opened a new log
I20240503 08:10:11.758237 2196 ts_tablet_manager.cc:1367] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:11.758631 2136 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd1d3f3edc54adbae4a580531a7b21b (DEFAULT_TABLE table=test-workload [id=d9845ce30f9e48eaba8cd996693c03db]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:11.758725 2136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd1d3f3edc54adbae4a580531a7b21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:11.758939 2196 raft_consensus.cc:348] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.759027 2196 raft_consensus.cc:374] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.759048 2196 raft_consensus.cc:729] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3acedb5ad844e60b5f7a0b72383a6b3, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.759112 2196 consensus_queue.cc:260] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.759291 2196 ts_tablet_manager.cc:1398] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:11.759368 2196 tablet_bootstrap.cc:492] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3: Bootstrap starting.
I20240503 08:10:11.759618 2197 tablet_bootstrap.cc:492] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708: No bootstrap required, opened a new log
I20240503 08:10:11.759672 2197 ts_tablet_manager.cc:1367] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:10:11.759739 2196 tablet_bootstrap.cc:654] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.759745 2195 tablet_bootstrap.cc:492] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b: No bootstrap required, opened a new log
I20240503 08:10:11.759815 2195 ts_tablet_manager.cc:1367] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:10:11.759838 2197 raft_consensus.cc:348] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.759891 2197 raft_consensus.cc:374] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.759908 2197 raft_consensus.cc:729] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe2aae583d14ad2a6892ae954d49708, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.759949 2197 consensus_queue.cc:260] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.759958 2195 raft_consensus.cc:348] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.760017 2195 raft_consensus.cc:374] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.760084 2197 ts_tablet_manager.cc:1398] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:11.760267 2197 tablet_bootstrap.cc:492] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708: Bootstrap starting.
I20240503 08:10:11.760622 2197 tablet_bootstrap.cc:654] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.760910 2195 raft_consensus.cc:729] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c5510f4594a3e96c849a3c203118b, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.761103 2197 tablet_bootstrap.cc:492] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708: No bootstrap required, opened a new log
I20240503 08:10:11.761139 2197 ts_tablet_manager.cc:1367] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:11.761255 2197 raft_consensus.cc:348] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.761309 2197 raft_consensus.cc:374] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.761320 2197 raft_consensus.cc:729] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe2aae583d14ad2a6892ae954d49708, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.761363 2197 consensus_queue.cc:260] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.761524 2197 ts_tablet_manager.cc:1398] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:11.761587 2197 tablet_bootstrap.cc:492] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708: Bootstrap starting.
I20240503 08:10:11.761916 2197 tablet_bootstrap.cc:654] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.762221 2195 consensus_queue.cc:260] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.762396 2195 ts_tablet_manager.cc:1398] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 08:10:11.762459 2195 tablet_bootstrap.cc:492] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b: Bootstrap starting.
I20240503 08:10:11.762480 2197 tablet_bootstrap.cc:492] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708: No bootstrap required, opened a new log
I20240503 08:10:11.762532 2197 ts_tablet_manager.cc:1367] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:11.762682 2197 raft_consensus.cc:348] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.762744 2197 raft_consensus.cc:374] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.762765 2197 raft_consensus.cc:729] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe2aae583d14ad2a6892ae954d49708, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.762828 2197 consensus_queue.cc:260] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.762858 2195 tablet_bootstrap.cc:654] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.762917 2197 ts_tablet_manager.cc:1398] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:11.762969 2197 tablet_bootstrap.cc:492] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708: Bootstrap starting.
I20240503 08:10:11.763394 2197 tablet_bootstrap.cc:654] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.763485 2196 tablet_bootstrap.cc:492] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3: No bootstrap required, opened a new log
I20240503 08:10:11.763526 2196 ts_tablet_manager.cc:1367] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:10:11.763610 2195 tablet_bootstrap.cc:492] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b: No bootstrap required, opened a new log
I20240503 08:10:11.763648 2195 ts_tablet_manager.cc:1367] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:11.763665 2196 raft_consensus.cc:348] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.763720 2196 raft_consensus.cc:374] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.763737 2196 raft_consensus.cc:729] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3acedb5ad844e60b5f7a0b72383a6b3, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.763754 2195 raft_consensus.cc:348] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.763808 2196 consensus_queue.cc:260] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.763832 2195 raft_consensus.cc:374] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.763937 2195 raft_consensus.cc:729] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c5510f4594a3e96c849a3c203118b, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.763962 2196 ts_tablet_manager.cc:1398] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:11.763986 2197 tablet_bootstrap.cc:492] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708: No bootstrap required, opened a new log
I20240503 08:10:11.764034 2195 consensus_queue.cc:260] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.764082 2196 tablet_bootstrap.cc:492] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3: Bootstrap starting.
I20240503 08:10:11.764086 2197 ts_tablet_manager.cc:1367] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:11.764691 2197 raft_consensus.cc:348] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.764756 2197 raft_consensus.cc:374] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.764775 2197 raft_consensus.cc:729] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe2aae583d14ad2a6892ae954d49708, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.764819 2197 consensus_queue.cc:260] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.764902 2197 ts_tablet_manager.cc:1398] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:11.765228 2196 tablet_bootstrap.cc:654] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.764250 2195 ts_tablet_manager.cc:1398] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:11.765594 2195 tablet_bootstrap.cc:492] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b: Bootstrap starting.
I20240503 08:10:11.765786 2196 tablet_bootstrap.cc:492] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3: No bootstrap required, opened a new log
I20240503 08:10:11.765839 2196 ts_tablet_manager.cc:1367] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:11.765966 2195 tablet_bootstrap.cc:654] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.765995 2196 raft_consensus.cc:348] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.766057 2196 raft_consensus.cc:374] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.766083 2196 raft_consensus.cc:729] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3acedb5ad844e60b5f7a0b72383a6b3, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.766127 2196 consensus_queue.cc:260] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.766232 2196 ts_tablet_manager.cc:1398] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:11.766305 2196 tablet_bootstrap.cc:492] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3: Bootstrap starting.
I20240503 08:10:11.766659 2196 tablet_bootstrap.cc:654] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.766762 2195 tablet_bootstrap.cc:492] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b: No bootstrap required, opened a new log
I20240503 08:10:11.766824 2195 ts_tablet_manager.cc:1367] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:11.766953 2195 raft_consensus.cc:348] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.766999 2195 raft_consensus.cc:374] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.767021 2195 raft_consensus.cc:729] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c5510f4594a3e96c849a3c203118b, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.767074 2195 consensus_queue.cc:260] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.767208 2195 ts_tablet_manager.cc:1398] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:11.767261 2196 tablet_bootstrap.cc:492] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3: No bootstrap required, opened a new log
I20240503 08:10:11.767300 2195 tablet_bootstrap.cc:492] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b: Bootstrap starting.
I20240503 08:10:11.767303 2196 ts_tablet_manager.cc:1367] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:11.767457 2196 raft_consensus.cc:348] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.767575 2196 raft_consensus.cc:374] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.767611 2196 raft_consensus.cc:729] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3acedb5ad844e60b5f7a0b72383a6b3, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.767652 2195 tablet_bootstrap.cc:654] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:11.767670 2196 consensus_queue.cc:260] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.767853 2196 ts_tablet_manager.cc:1398] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:11.768146 2195 tablet_bootstrap.cc:492] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b: No bootstrap required, opened a new log
I20240503 08:10:11.768193 2195 ts_tablet_manager.cc:1367] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:11.768366 2195 raft_consensus.cc:348] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.768452 2195 raft_consensus.cc:374] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:11.768471 2195 raft_consensus.cc:729] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c5510f4594a3e96c849a3c203118b, State: Initialized, Role: FOLLOWER
I20240503 08:10:11.768501 2195 consensus_queue.cc:260] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.768590 2195 ts_tablet_manager.cc:1398] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:11.834486 2207 raft_consensus.cc:482] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:11.834627 2207 raft_consensus.cc:504] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.834905 2207 leader_election.cc:288] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521), 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:11.837657 2149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4416cf3a68f546e49bf05e9e74b08773" candidate_uuid: "685c5510f4594a3e96c849a3c203118b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe2aae583d14ad2a6892ae954d49708" is_pre_election: true
I20240503 08:10:11.837664 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4416cf3a68f546e49bf05e9e74b08773" candidate_uuid: "685c5510f4594a3e96c849a3c203118b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" is_pre_election: true
I20240503 08:10:11.837785 2149 raft_consensus.cc:2454] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c5510f4594a3e96c849a3c203118b in term 0.
I20240503 08:10:11.837810 2074 raft_consensus.cc:2454] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c5510f4594a3e96c849a3c203118b in term 0.
I20240503 08:10:11.837934 1961 leader_election.cc:302] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, 685c5510f4594a3e96c849a3c203118b; no voters:
I20240503 08:10:11.838029 2207 raft_consensus.cc:2781] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:11.838074 2207 raft_consensus.cc:482] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:11.838088 2207 raft_consensus.cc:3037] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.838784 2207 raft_consensus.cc:504] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.838979 2207 leader_election.cc:288] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [CANDIDATE]: Term 1 election: Requested vote from peers f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521), 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:11.839138 2149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4416cf3a68f546e49bf05e9e74b08773" candidate_uuid: "685c5510f4594a3e96c849a3c203118b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe2aae583d14ad2a6892ae954d49708"
I20240503 08:10:11.839190 2149 raft_consensus.cc:3037] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.839300 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4416cf3a68f546e49bf05e9e74b08773" candidate_uuid: "685c5510f4594a3e96c849a3c203118b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3"
I20240503 08:10:11.839365 2074 raft_consensus.cc:3037] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.839829 2149 raft_consensus.cc:2454] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c5510f4594a3e96c849a3c203118b in term 1.
I20240503 08:10:11.840019 2074 raft_consensus.cc:2454] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c5510f4594a3e96c849a3c203118b in term 1.
I20240503 08:10:11.839990 1961 leader_election.cc:302] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, 685c5510f4594a3e96c849a3c203118b; no voters:
I20240503 08:10:11.840155 2207 raft_consensus.cc:2781] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:11.840340 2207 raft_consensus.cc:686] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 1 LEADER]: Becoming Leader. State: Replica: 685c5510f4594a3e96c849a3c203118b, State: Running, Role: LEADER
I20240503 08:10:11.840400 2207 consensus_queue.cc:237] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.840965 1894 catalog_manager.cc:5484] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c5510f4594a3e96c849a3c203118b (127.28.55.65). New cstate: current_term: 1 leader_uuid: "685c5510f4594a3e96c849a3c203118b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:11.886502 2200 raft_consensus.cc:482] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:11.886627 2200 raft_consensus.cc:504] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.886902 2200 leader_election.cc:288] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523), 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:11.890650 2149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a662efda5fc4f92a84284a3ffb1ef4b" candidate_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe2aae583d14ad2a6892ae954d49708" is_pre_election: true
I20240503 08:10:11.890662 1998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a662efda5fc4f92a84284a3ffb1ef4b" candidate_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c5510f4594a3e96c849a3c203118b" is_pre_election: true
I20240503 08:10:11.890775 2149 raft_consensus.cc:2454] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3acedb5ad844e60b5f7a0b72383a6b3 in term 0.
I20240503 08:10:11.890794 1998 raft_consensus.cc:2454] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3acedb5ad844e60b5f7a0b72383a6b3 in term 0.
I20240503 08:10:11.890938 2037 leader_election.cc:302] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, f3acedb5ad844e60b5f7a0b72383a6b3; no voters:
I20240503 08:10:11.891038 2200 raft_consensus.cc:2781] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:11.891084 2200 raft_consensus.cc:482] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:11.891098 2200 raft_consensus.cc:3037] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.893819 2200 raft_consensus.cc:504] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.893985 2200 leader_election.cc:288] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 1 election: Requested vote from peers 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523), 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:11.894163 1998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a662efda5fc4f92a84284a3ffb1ef4b" candidate_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c5510f4594a3e96c849a3c203118b"
I20240503 08:10:11.894163 2149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a662efda5fc4f92a84284a3ffb1ef4b" candidate_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe2aae583d14ad2a6892ae954d49708"
I20240503 08:10:11.894244 1998 raft_consensus.cc:3037] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.894253 2149 raft_consensus.cc:3037] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.895037 2149 raft_consensus.cc:2454] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3acedb5ad844e60b5f7a0b72383a6b3 in term 1.
I20240503 08:10:11.895037 1998 raft_consensus.cc:2454] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3acedb5ad844e60b5f7a0b72383a6b3 in term 1.
I20240503 08:10:11.895241 2037 leader_election.cc:302] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, f3acedb5ad844e60b5f7a0b72383a6b3; no voters:
I20240503 08:10:11.895385 2200 raft_consensus.cc:2781] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:11.895546 2200 raft_consensus.cc:686] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 LEADER]: Becoming Leader. State: Replica: f3acedb5ad844e60b5f7a0b72383a6b3, State: Running, Role: LEADER
I20240503 08:10:11.895601 2200 consensus_queue.cc:237] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.896171 1894 catalog_manager.cc:5484] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66). New cstate: current_term: 1 leader_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:11.925202 2203 raft_consensus.cc:482] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:11.925323 2203 raft_consensus.cc:504] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.925565 2203 leader_election.cc:288] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523), f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521)
I20240503 08:10:11.928269 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd1d3f3edc54adbae4a580531a7b21b" candidate_uuid: "3fe2aae583d14ad2a6892ae954d49708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" is_pre_election: true
I20240503 08:10:11.928269 1998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd1d3f3edc54adbae4a580531a7b21b" candidate_uuid: "3fe2aae583d14ad2a6892ae954d49708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c5510f4594a3e96c849a3c203118b" is_pre_election: true
I20240503 08:10:11.928416 2074 raft_consensus.cc:2454] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fe2aae583d14ad2a6892ae954d49708 in term 0.
I20240503 08:10:11.928442 1998 raft_consensus.cc:2454] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fe2aae583d14ad2a6892ae954d49708 in term 0.
I20240503 08:10:11.928608 2113 leader_election.cc:302] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, 685c5510f4594a3e96c849a3c203118b; no voters:
I20240503 08:10:11.928716 2203 raft_consensus.cc:2781] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:11.928766 2203 raft_consensus.cc:482] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:11.928779 2203 raft_consensus.cc:3037] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.929448 2203 raft_consensus.cc:504] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.929579 2203 leader_election.cc:288] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 1 election: Requested vote from peers 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523), f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521)
I20240503 08:10:11.929752 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd1d3f3edc54adbae4a580531a7b21b" candidate_uuid: "3fe2aae583d14ad2a6892ae954d49708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3"
I20240503 08:10:11.929806 2074 raft_consensus.cc:3037] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.929773 1998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd1d3f3edc54adbae4a580531a7b21b" candidate_uuid: "3fe2aae583d14ad2a6892ae954d49708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c5510f4594a3e96c849a3c203118b"
I20240503 08:10:11.929837 1998 raft_consensus.cc:3037] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.930431 2074 raft_consensus.cc:2454] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fe2aae583d14ad2a6892ae954d49708 in term 1.
I20240503 08:10:11.930480 1998 raft_consensus.cc:2454] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fe2aae583d14ad2a6892ae954d49708 in term 1.
I20240503 08:10:11.930588 2114 leader_election.cc:302] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, f3acedb5ad844e60b5f7a0b72383a6b3; no voters:
I20240503 08:10:11.930693 2203 raft_consensus.cc:2781] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:11.930747 2203 raft_consensus.cc:686] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 LEADER]: Becoming Leader. State: Replica: 3fe2aae583d14ad2a6892ae954d49708, State: Running, Role: LEADER
I20240503 08:10:11.930824 2203 consensus_queue.cc:237] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.931453 1894 catalog_manager.cc:5484] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67). New cstate: current_term: 1 leader_uuid: "3fe2aae583d14ad2a6892ae954d49708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:11.954597 2207 raft_consensus.cc:482] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:11.954726 2207 raft_consensus.cc:504] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.954944 2207 leader_election.cc:288] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521), 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:11.955176 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cafb41bf92a4f36877bf48362aee7a1" candidate_uuid: "685c5510f4594a3e96c849a3c203118b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" is_pre_election: true
I20240503 08:10:11.955245 2074 raft_consensus.cc:2454] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c5510f4594a3e96c849a3c203118b in term 0.
I20240503 08:10:11.955181 2149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cafb41bf92a4f36877bf48362aee7a1" candidate_uuid: "685c5510f4594a3e96c849a3c203118b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe2aae583d14ad2a6892ae954d49708" is_pre_election: true
I20240503 08:10:11.955296 2149 raft_consensus.cc:2454] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c5510f4594a3e96c849a3c203118b in term 0.
I20240503 08:10:11.955397 1963 leader_election.cc:302] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c5510f4594a3e96c849a3c203118b, f3acedb5ad844e60b5f7a0b72383a6b3; no voters:
I20240503 08:10:11.955471 2207 raft_consensus.cc:2781] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:11.955509 2207 raft_consensus.cc:482] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:11.955536 2207 raft_consensus.cc:3037] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.956190 2207 raft_consensus.cc:504] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.956332 2207 leader_election.cc:288] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [CANDIDATE]: Term 1 election: Requested vote from peers f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521), 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:11.956455 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cafb41bf92a4f36877bf48362aee7a1" candidate_uuid: "685c5510f4594a3e96c849a3c203118b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3"
I20240503 08:10:11.956473 2149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cafb41bf92a4f36877bf48362aee7a1" candidate_uuid: "685c5510f4594a3e96c849a3c203118b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe2aae583d14ad2a6892ae954d49708"
I20240503 08:10:11.956516 2074 raft_consensus.cc:3037] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.956529 2149 raft_consensus.cc:3037] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:11.957043 2074 raft_consensus.cc:2454] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c5510f4594a3e96c849a3c203118b in term 1.
I20240503 08:10:11.957093 2149 raft_consensus.cc:2454] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c5510f4594a3e96c849a3c203118b in term 1.
I20240503 08:10:11.957201 1963 leader_election.cc:302] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c5510f4594a3e96c849a3c203118b, f3acedb5ad844e60b5f7a0b72383a6b3; no voters:
I20240503 08:10:11.957284 2207 raft_consensus.cc:2781] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:11.957433 2207 raft_consensus.cc:686] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 1 LEADER]: Becoming Leader. State: Replica: 685c5510f4594a3e96c849a3c203118b, State: Running, Role: LEADER
I20240503 08:10:11.957604 2207 consensus_queue.cc:237] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:11.958233 1894 catalog_manager.cc:5484] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c5510f4594a3e96c849a3c203118b (127.28.55.65). New cstate: current_term: 1 leader_uuid: "685c5510f4594a3e96c849a3c203118b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:11.975719 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:10:11.978425 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:11.978583 28893 tablet_replica.cc:330] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b: stopping tablet replica
I20240503 08:10:11.978670 28893 raft_consensus.cc:2229] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:11.978717 28893 raft_consensus.cc:2258] T 1dd1d3f3edc54adbae4a580531a7b21b P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:11.979001 28893 tablet_replica.cc:330] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b: stopping tablet replica
I20240503 08:10:11.979048 28893 raft_consensus.cc:2229] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:11.979094 28893 pending_rounds.cc:62] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b: Trying to abort 1 pending ops.
I20240503 08:10:11.979117 28893 pending_rounds.cc:68] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508733778939904 op_type: NO_OP noop_request { }
I20240503 08:10:11.979176 28893 raft_consensus.cc:2866] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:10:11.979203 28893 raft_consensus.cc:2258] T 8cafb41bf92a4f36877bf48362aee7a1 P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:11.979456 28893 tablet_replica.cc:330] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b: stopping tablet replica
I20240503 08:10:11.979501 28893 raft_consensus.cc:2229] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:11.979535 28893 raft_consensus.cc:2258] T 5a662efda5fc4f92a84284a3ffb1ef4b P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:11.979692 28893 tablet_replica.cc:330] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b: stopping tablet replica
I20240503 08:10:11.979725 28893 raft_consensus.cc:2229] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:11.979753 28893 pending_rounds.cc:62] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b: Trying to abort 1 pending ops.
I20240503 08:10:11.979775 28893 pending_rounds.cc:68] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508733298831360 op_type: NO_OP noop_request { }
I20240503 08:10:11.979816 28893 raft_consensus.cc:2866] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:10:11.979851 28893 raft_consensus.cc:2258] T 4416cf3a68f546e49bf05e9e74b08773 P 685c5510f4594a3e96c849a3c203118b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:11.992089 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:10:12.398133 2226 consensus_queue.cc:1035] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:12.400077 2038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:10:12.400609 2038 consensus_peers.cc:485] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:12.449882 2203 consensus_queue.cc:1035] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:12.451787 2113 consensus_peers.cc:485] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:12.955597 2235 consensus_queue.cc:579] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Leader has been unable to successfully communicate with peer 685c5510f4594a3e96c849a3c203118b for more than 1 seconds (1.025s)
W20240503 08:10:12.956418 1895 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1dd1d3f3edc54adbae4a580531a7b21b with cas_config_opid_index -1: no extra replica candidate found for tablet 1dd1d3f3edc54adbae4a580531a7b21b (table test-workload [id=d9845ce30f9e48eaba8cd996693c03db]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20240503 08:10:12.993599 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:12.994990 2243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:12.995090 2245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:12.995143 2242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:12.995275 28893 server_base.cc:1034] running on GCE node
I20240503 08:10:12.995368 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:12.995393 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:12.995404 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723812995405 us; error 0 us; skew 500 ppm
I20240503 08:10:12.996068 28893 webserver.cc:460] Webserver started at http://127.28.55.68:39789/ using document root <none> and password file <none>
I20240503 08:10:12.996143 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:12.996181 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:12.996217 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:12.996487 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-3-root/instance:
uuid: "8f19408d91a24816b660335dfbd0685a"
format_stamp: "Formatted at 2024-05-03 08:10:12 on dist-test-slave-mlgg"
I20240503 08:10:12.997371 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:12.997853 2250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:12.997967 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:12.998010 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-3-root
uuid: "8f19408d91a24816b660335dfbd0685a"
format_stamp: "Formatted at 2024-05-03 08:10:12 on dist-test-slave-mlgg"
I20240503 08:10:12.998054 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714723660874102-28893-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:13.004139 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:13.004343 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:13.004501 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:13.004700 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:13.004727 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:13.004756 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:13.004781 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:13.007465 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.68:44611
I20240503 08:10:13.008601 2313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.68:44611 every 8 connection(s)
I20240503 08:10:13.011696 2314 heartbeater.cc:347] Connected to a master server at 127.28.55.126:45797
I20240503 08:10:13.011762 2314 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:13.011862 2314 heartbeater.cc:511] Master 127.28.55.126:45797 requested a full tablet report, sending...
I20240503 08:10:13.012094 1895 ts_manager.cc:194] Registered new tserver with Master: 8f19408d91a24816b660335dfbd0685a (127.28.55.68:44611)
I20240503 08:10:13.012421 1895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56998
I20240503 08:10:13.044761 2149 consensus_queue.cc:237] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:13.045419 2074 raft_consensus.cc:1261] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:13.045680 2238 consensus_queue.cc:1035] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:10:13.045858 2113 consensus_peers.cc:485] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:13.046281 2239 raft_consensus.cc:2932] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } } }
I20240503 08:10:13.046427 2074 raft_consensus.cc:2932] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } } }
I20240503 08:10:13.047475 1879 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1dd1d3f3edc54adbae4a580531a7b21b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
I20240503 08:10:13.047673 1895 catalog_manager.cc:5484] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 reported cstate change: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "3fe2aae583d14ad2a6892ae954d49708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:10:13.048887 2115 consensus_peers.cc:485] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 -> Peer 8f19408d91a24816b660335dfbd0685a (127.28.55.68:44611): Couldn't send request to peer 8f19408d91a24816b660335dfbd0685a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dd1d3f3edc54adbae4a580531a7b21b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:13.345485 2226 consensus_queue.cc:579] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Leader has been unable to successfully communicate with peer 685c5510f4594a3e96c849a3c203118b for more than 1 seconds (1.450s)
I20240503 08:10:13.346521 2074 consensus_queue.cc:237] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:13.347036 2149 raft_consensus.cc:1261] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:13.347298 2327 consensus_queue.cc:1035] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:10:13.347471 2038 consensus_peers.cc:485] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:13.348205 2226 raft_consensus.cc:2932] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } } }
I20240503 08:10:13.348346 2149 raft_consensus.cc:2932] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } } }
I20240503 08:10:13.349452 1881 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5a662efda5fc4f92a84284a3ffb1ef4b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:10:13.349623 1894 catalog_manager.cc:5484] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 reported cstate change: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:10:13.349937 2040 consensus_peers.cc:485] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 -> Peer 8f19408d91a24816b660335dfbd0685a (127.28.55.68:44611): Couldn't send request to peer 8f19408d91a24816b660335dfbd0685a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a662efda5fc4f92a84284a3ffb1ef4b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:13.394497 2324 raft_consensus.cc:482] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:13.394634 2324 raft_consensus.cc:504] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:13.394860 2324 leader_election.cc:288] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523), 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:13.395099 2149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4416cf3a68f546e49bf05e9e74b08773" candidate_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe2aae583d14ad2a6892ae954d49708" is_pre_election: true
I20240503 08:10:13.395172 2149 raft_consensus.cc:2454] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3acedb5ad844e60b5f7a0b72383a6b3 in term 1.
I20240503 08:10:13.395303 2037 leader_election.cc:302] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, f3acedb5ad844e60b5f7a0b72383a6b3; no voters:
I20240503 08:10:13.395397 2324 raft_consensus.cc:2781] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:10:13.395459 2324 raft_consensus.cc:482] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 08:10:13.395470 2038 leader_election.cc:334] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
I20240503 08:10:13.395488 2324 raft_consensus.cc:3037] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:13.396193 2324 raft_consensus.cc:504] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:13.396314 2324 leader_election.cc:288] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 2 election: Requested vote from peers 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523), 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:13.396467 2149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4416cf3a68f546e49bf05e9e74b08773" candidate_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe2aae583d14ad2a6892ae954d49708"
I20240503 08:10:13.396531 2149 raft_consensus.cc:3037] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:10:13.396806 2038 leader_election.cc:334] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
I20240503 08:10:13.397153 2149 raft_consensus.cc:2454] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3acedb5ad844e60b5f7a0b72383a6b3 in term 2.
I20240503 08:10:13.397341 2037 leader_election.cc:302] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, f3acedb5ad844e60b5f7a0b72383a6b3; no voters: 685c5510f4594a3e96c849a3c203118b
I20240503 08:10:13.397426 2324 raft_consensus.cc:2781] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:13.397481 2324 raft_consensus.cc:686] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 LEADER]: Becoming Leader. State: Replica: f3acedb5ad844e60b5f7a0b72383a6b3, State: Running, Role: LEADER
I20240503 08:10:13.397529 2324 consensus_queue.cc:237] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:13.398133 1894 catalog_manager.cc:5484] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 reported cstate change: term changed from 1 to 2, leader changed from 685c5510f4594a3e96c849a3c203118b (127.28.55.65) to f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66). New cstate: current_term: 2 leader_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:13.553432 2331 ts_tablet_manager.cc:902] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: Initiating tablet copy from peer 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:13.553750 2331 tablet_copy_client.cc:308] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.67:45635
I20240503 08:10:13.556427 2159 tablet_copy_service.cc:140] P 3fe2aae583d14ad2a6892ae954d49708: Received BeginTabletCopySession request for tablet 1dd1d3f3edc54adbae4a580531a7b21b from peer 8f19408d91a24816b660335dfbd0685a ({username='slave'} at 127.0.0.1:37608)
I20240503 08:10:13.556501 2159 tablet_copy_service.cc:161] P 3fe2aae583d14ad2a6892ae954d49708: Beginning new tablet copy session on tablet 1dd1d3f3edc54adbae4a580531a7b21b from peer 8f19408d91a24816b660335dfbd0685a at {username='slave'} at 127.0.0.1:37608: session id = 8f19408d91a24816b660335dfbd0685a-1dd1d3f3edc54adbae4a580531a7b21b
I20240503 08:10:13.556990 2159 tablet_copy_source_session.cc:205] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:13.557296 2331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd1d3f3edc54adbae4a580531a7b21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:13.557894 2238 raft_consensus.cc:482] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:13.557966 2238 raft_consensus.cc:504] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:13.558128 2238 leader_election.cc:288] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523), f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521)
I20240503 08:10:13.558383 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cafb41bf92a4f36877bf48362aee7a1" candidate_uuid: "3fe2aae583d14ad2a6892ae954d49708" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" is_pre_election: true
I20240503 08:10:13.558452 2331 tablet_copy_client.cc:783] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:13.558459 2074 raft_consensus.cc:2454] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fe2aae583d14ad2a6892ae954d49708 in term 1.
I20240503 08:10:13.558568 2331 tablet_copy_client.cc:647] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:13.558617 2114 leader_election.cc:302] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, f3acedb5ad844e60b5f7a0b72383a6b3; no voters:
I20240503 08:10:13.558703 2238 raft_consensus.cc:2781] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:10:13.558743 2238 raft_consensus.cc:482] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:13.558771 2238 raft_consensus.cc:3037] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:10:13.558795 2113 leader_election.cc:334] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
I20240503 08:10:13.559154 2331 tablet_copy_client.cc:523] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:13.559446 2238 raft_consensus.cc:504] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:13.559559 2238 leader_election.cc:288] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 2 election: Requested vote from peers 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523), f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521)
I20240503 08:10:13.559725 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cafb41bf92a4f36877bf48362aee7a1" candidate_uuid: "3fe2aae583d14ad2a6892ae954d49708" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3"
I20240503 08:10:13.559779 2331 tablet_bootstrap.cc:492] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: Bootstrap starting.
I20240503 08:10:13.559794 2074 raft_consensus.cc:3037] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:10:13.559988 2113 leader_election.cc:334] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
I20240503 08:10:13.560518 2074 raft_consensus.cc:2454] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fe2aae583d14ad2a6892ae954d49708 in term 2.
I20240503 08:10:13.560676 2114 leader_election.cc:302] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fe2aae583d14ad2a6892ae954d49708, f3acedb5ad844e60b5f7a0b72383a6b3; no voters: 685c5510f4594a3e96c849a3c203118b
I20240503 08:10:13.560782 2238 raft_consensus.cc:2781] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:13.560901 2238 raft_consensus.cc:686] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 LEADER]: Becoming Leader. State: Replica: 3fe2aae583d14ad2a6892ae954d49708, State: Running, Role: LEADER
I20240503 08:10:13.560968 2238 consensus_queue.cc:237] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } }
I20240503 08:10:13.561025 2331 tablet_bootstrap.cc:492] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:13.561199 2331 tablet_bootstrap.cc:492] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: Bootstrap complete.
I20240503 08:10:13.561295 2331 ts_tablet_manager.cc:1367] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:10:13.561422 2331 raft_consensus.cc:348] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:13.561491 2331 raft_consensus.cc:729] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f19408d91a24816b660335dfbd0685a, State: Initialized, Role: LEARNER
I20240503 08:10:13.561456 1894 catalog_manager.cc:5484] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 reported cstate change: term changed from 1 to 2, leader changed from 685c5510f4594a3e96c849a3c203118b (127.28.55.65) to 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67). New cstate: current_term: 2 leader_uuid: "3fe2aae583d14ad2a6892ae954d49708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:13.561519 2331 consensus_queue.cc:260] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:13.561671 2331 ts_tablet_manager.cc:1398] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:13.561704 2314 heartbeater.cc:503] Master 127.28.55.126:45797 was elected leader, sending a full tablet report...
I20240503 08:10:13.561975 2159 tablet_copy_service.cc:342] P 3fe2aae583d14ad2a6892ae954d49708: Request end of tablet copy session 8f19408d91a24816b660335dfbd0685a-1dd1d3f3edc54adbae4a580531a7b21b received from {username='slave'} at 127.0.0.1:37608
I20240503 08:10:13.562021 2159 tablet_copy_service.cc:433] P 3fe2aae583d14ad2a6892ae954d49708: ending tablet copy session 8f19408d91a24816b660335dfbd0685a-1dd1d3f3edc54adbae4a580531a7b21b on tablet 1dd1d3f3edc54adbae4a580531a7b21b with peer 8f19408d91a24816b660335dfbd0685a
W20240503 08:10:13.644259 1944 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 685c5510f4594a3e96c849a3c203118b not available for placement
I20240503 08:10:13.840597 2331 ts_tablet_manager.cc:902] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: Initiating tablet copy from peer f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521)
I20240503 08:10:13.840947 2331 tablet_copy_client.cc:308] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:37521
I20240503 08:10:13.843928 2084 tablet_copy_service.cc:140] P f3acedb5ad844e60b5f7a0b72383a6b3: Received BeginTabletCopySession request for tablet 5a662efda5fc4f92a84284a3ffb1ef4b from peer 8f19408d91a24816b660335dfbd0685a ({username='slave'} at 127.0.0.1:58586)
I20240503 08:10:13.844017 2084 tablet_copy_service.cc:161] P f3acedb5ad844e60b5f7a0b72383a6b3: Beginning new tablet copy session on tablet 5a662efda5fc4f92a84284a3ffb1ef4b from peer 8f19408d91a24816b660335dfbd0685a at {username='slave'} at 127.0.0.1:58586: session id = 8f19408d91a24816b660335dfbd0685a-5a662efda5fc4f92a84284a3ffb1ef4b
I20240503 08:10:13.844583 2084 tablet_copy_source_session.cc:205] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:13.844954 2331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a662efda5fc4f92a84284a3ffb1ef4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:13.845919 2331 tablet_copy_client.cc:783] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:13.846036 2331 tablet_copy_client.cc:647] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:13.846549 2331 tablet_copy_client.cc:523] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:13.847179 2331 tablet_bootstrap.cc:492] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: Bootstrap starting.
I20240503 08:10:13.848208 2331 tablet_bootstrap.cc:492] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:13.848351 2331 tablet_bootstrap.cc:492] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: Bootstrap complete.
I20240503 08:10:13.848420 2331 ts_tablet_manager.cc:1367] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:13.848521 2331 raft_consensus.cc:348] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:13.848582 2331 raft_consensus.cc:729] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f19408d91a24816b660335dfbd0685a, State: Initialized, Role: LEARNER
I20240503 08:10:13.848601 2331 consensus_queue.cc:260] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:13.848682 2331 ts_tablet_manager.cc:1398] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:13.848908 2084 tablet_copy_service.cc:342] P f3acedb5ad844e60b5f7a0b72383a6b3: Request end of tablet copy session 8f19408d91a24816b660335dfbd0685a-5a662efda5fc4f92a84284a3ffb1ef4b received from {username='slave'} at 127.0.0.1:58586
I20240503 08:10:13.848963 2084 tablet_copy_service.cc:433] P f3acedb5ad844e60b5f7a0b72383a6b3: ending tablet copy session 8f19408d91a24816b660335dfbd0685a-5a662efda5fc4f92a84284a3ffb1ef4b on tablet 5a662efda5fc4f92a84284a3ffb1ef4b with peer 8f19408d91a24816b660335dfbd0685a
W20240503 08:10:13.870407 2038 consensus_peers.cc:485] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:13.885195 2149 raft_consensus.cc:1261] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 08:10:13.885627 2324 consensus_queue.cc:1035] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:13.969055 2288 raft_consensus.cc:1203] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:10:13.979749 2074 raft_consensus.cc:1261] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 08:10:13.980082 2339 consensus_queue.cc:1035] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:13.981604 2113 consensus_peers.cc:485] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:14.095255 2352 consensus_queue.cc:579] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Leader has been unable to successfully communicate with peer 685c5510f4594a3e96c849a3c203118b for more than 1 seconds (1.050s)
I20240503 08:10:14.237967 2288 raft_consensus.cc:1203] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:10:14.429924 2352 raft_consensus.cc:1050] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708: attempting to promote NON_VOTER 8f19408d91a24816b660335dfbd0685a to VOTER
I20240503 08:10:14.430155 2352 consensus_queue.cc:237] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } }
I20240503 08:10:14.430667 2288 raft_consensus.cc:1261] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [term 1 LEARNER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:14.430692 2074 raft_consensus.cc:1261] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:14.430900 2355 consensus_queue.cc:1035] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:14.431025 2352 consensus_queue.cc:1035] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 08:10:14.431143 2113 consensus_peers.cc:485] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:14.431558 2353 consensus_queue.cc:579] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Leader has been unable to successfully communicate with peer 685c5510f4594a3e96c849a3c203118b for more than 1 seconds (1.085s)
I20240503 08:10:14.431782 2352 raft_consensus.cc:2932] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.431901 2288 raft_consensus.cc:2932] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.432984 2074 raft_consensus.cc:2932] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.433079 1893 catalog_manager.cc:5484] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a reported cstate change: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fe2aae583d14ad2a6892ae954d49708" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.434341 2149 consensus_queue.cc:237] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } }
I20240503 08:10:14.434695 2074 raft_consensus.cc:1261] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:10:14.434725 2288 raft_consensus.cc:1261] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:10:14.434891 2238 consensus_queue.cc:1035] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:14.434981 2355 consensus_queue.cc:1035] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:14.435453 2339 raft_consensus.cc:2932] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.435567 2288 raft_consensus.cc:2932] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.435586 2074 raft_consensus.cc:2932] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.436517 1879 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1dd1d3f3edc54adbae4a580531a7b21b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:14.436527 1893 catalog_manager.cc:5484] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a reported cstate change: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New cstate: current_term: 1 leader_uuid: "3fe2aae583d14ad2a6892ae954d49708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
W20240503 08:10:14.437719 1880 catalog_manager.cc:4573] TS 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1dd1d3f3edc54adbae4a580531a7b21b: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
I20240503 08:10:14.439469 2324 consensus_queue.cc:579] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Leader has been unable to successfully communicate with peer 685c5510f4594a3e96c849a3c203118b for more than 1 seconds (1.042s)
I20240503 08:10:14.440138 2074 consensus_queue.cc:237] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:14.440536 2149 raft_consensus.cc:1261] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 08:10:14.440580 2040 consensus_peers.cc:485] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 -> Peer 8f19408d91a24816b660335dfbd0685a (127.28.55.68:44611): Couldn't send request to peer 8f19408d91a24816b660335dfbd0685a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4416cf3a68f546e49bf05e9e74b08773. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:14.440706 2353 consensus_queue.cc:1035] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:10:14.440927 2038 consensus_peers.cc:485] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:14.441340 2327 raft_consensus.cc:2932] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } } }
I20240503 08:10:14.441498 2149 raft_consensus.cc:2932] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } } }
I20240503 08:10:14.442454 1881 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4416cf3a68f546e49bf05e9e74b08773 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:10:14.442636 1895 catalog_manager.cc:5484] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 reported cstate change: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New cstate: current_term: 2 leader_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:10:14.644564 1944 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 685c5510f4594a3e96c849a3c203118b not available for placement
I20240503 08:10:14.770438 2324 raft_consensus.cc:1050] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3: attempting to promote NON_VOTER 8f19408d91a24816b660335dfbd0685a to VOTER
I20240503 08:10:14.770634 2324 consensus_queue.cc:237] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } }
I20240503 08:10:14.771183 2288 raft_consensus.cc:1261] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [term 1 LEARNER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:14.771342 2149 raft_consensus.cc:1261] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:14.771427 2327 consensus_queue.cc:1035] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:14.771560 2327 consensus_queue.cc:1035] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 08:10:14.771728 2038 consensus_peers.cc:485] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:14.772323 2353 raft_consensus.cc:2932] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.772437 2149 raft_consensus.cc:2932] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.773578 1894 catalog_manager.cc:5484] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 reported cstate change: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.773831 2288 raft_consensus.cc:2932] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.774565 2074 consensus_queue.cc:237] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } }
I20240503 08:10:14.775108 2288 raft_consensus.cc:1261] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:10:14.775259 2149 raft_consensus.cc:1261] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:10:14.775285 2324 consensus_queue.cc:1035] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:14.775503 2324 consensus_queue.cc:1035] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:14.775859 2353 raft_consensus.cc:2932] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.776016 2288 raft_consensus.cc:2932] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.776297 2149 raft_consensus.cc:2932] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:14.776783 1881 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a662efda5fc4f92a84284a3ffb1ef4b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:14.777081 1895 catalog_manager.cc:5484] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a reported cstate change: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New cstate: current_term: 1 leader_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:15.032780 2355 consensus_queue.cc:579] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Leader has been unable to successfully communicate with peer 685c5510f4594a3e96c849a3c203118b for more than 1 seconds (1.472s)
I20240503 08:10:15.033821 2149 consensus_queue.cc:237] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:15.034360 2074 raft_consensus.cc:1261] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 08:10:15.034420 2115 consensus_peers.cc:485] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 -> Peer 8f19408d91a24816b660335dfbd0685a (127.28.55.68:44611): Couldn't send request to peer 8f19408d91a24816b660335dfbd0685a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cafb41bf92a4f36877bf48362aee7a1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:15.034546 2370 consensus_queue.cc:1035] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:10:15.034750 2113 consensus_peers.cc:485] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:15.035282 2369 raft_consensus.cc:2932] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } } }
I20240503 08:10:15.035410 2074 raft_consensus.cc:2932] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } } }
I20240503 08:10:15.036173 1879 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8cafb41bf92a4f36877bf48362aee7a1 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:10:15.036348 1894 catalog_manager.cc:5484] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 reported cstate change: config changed from index -1 to 2, NON_VOTER 8f19408d91a24816b660335dfbd0685a (127.28.55.68) added. New cstate: current_term: 2 leader_uuid: "3fe2aae583d14ad2a6892ae954d49708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } } }
I20240503 08:10:15.043612 2374 ts_tablet_manager.cc:902] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: Initiating tablet copy from peer f3acedb5ad844e60b5f7a0b72383a6b3 (127.28.55.66:37521)
I20240503 08:10:15.043956 2374 tablet_copy_client.cc:308] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:37521
I20240503 08:10:15.044171 2084 tablet_copy_service.cc:140] P f3acedb5ad844e60b5f7a0b72383a6b3: Received BeginTabletCopySession request for tablet 4416cf3a68f546e49bf05e9e74b08773 from peer 8f19408d91a24816b660335dfbd0685a ({username='slave'} at 127.0.0.1:58586)
I20240503 08:10:15.044212 2084 tablet_copy_service.cc:161] P f3acedb5ad844e60b5f7a0b72383a6b3: Beginning new tablet copy session on tablet 4416cf3a68f546e49bf05e9e74b08773 from peer 8f19408d91a24816b660335dfbd0685a at {username='slave'} at 127.0.0.1:58586: session id = 8f19408d91a24816b660335dfbd0685a-4416cf3a68f546e49bf05e9e74b08773
I20240503 08:10:15.044704 2084 tablet_copy_source_session.cc:205] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:15.045014 2374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4416cf3a68f546e49bf05e9e74b08773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:15.046036 2374 tablet_copy_client.cc:783] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:15.046137 2374 tablet_copy_client.cc:647] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:15.046638 2374 tablet_copy_client.cc:523] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:15.047276 2374 tablet_bootstrap.cc:492] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: Bootstrap starting.
I20240503 08:10:15.048278 2374 tablet_bootstrap.cc:492] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:15.048419 2374 tablet_bootstrap.cc:492] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: Bootstrap complete.
I20240503 08:10:15.048491 2374 ts_tablet_manager.cc:1367] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:15.048586 2374 raft_consensus.cc:348] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:15.048645 2374 raft_consensus.cc:729] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8f19408d91a24816b660335dfbd0685a, State: Initialized, Role: LEARNER
I20240503 08:10:15.048666 2374 consensus_queue.cc:260] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:15.048734 2374 ts_tablet_manager.cc:1398] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:15.048957 2084 tablet_copy_service.cc:342] P f3acedb5ad844e60b5f7a0b72383a6b3: Request end of tablet copy session 8f19408d91a24816b660335dfbd0685a-4416cf3a68f546e49bf05e9e74b08773 received from {username='slave'} at 127.0.0.1:58586
I20240503 08:10:15.049021 2084 tablet_copy_service.cc:433] P f3acedb5ad844e60b5f7a0b72383a6b3: ending tablet copy session 8f19408d91a24816b660335dfbd0685a-4416cf3a68f546e49bf05e9e74b08773 on tablet 4416cf3a68f546e49bf05e9e74b08773 with peer 8f19408d91a24816b660335dfbd0685a
I20240503 08:10:15.474087 2374 ts_tablet_manager.cc:902] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: Initiating tablet copy from peer 3fe2aae583d14ad2a6892ae954d49708 (127.28.55.67:45635)
I20240503 08:10:15.474380 2374 tablet_copy_client.cc:308] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.67:45635
I20240503 08:10:15.474620 2159 tablet_copy_service.cc:140] P 3fe2aae583d14ad2a6892ae954d49708: Received BeginTabletCopySession request for tablet 8cafb41bf92a4f36877bf48362aee7a1 from peer 8f19408d91a24816b660335dfbd0685a ({username='slave'} at 127.0.0.1:37608)
I20240503 08:10:15.474671 2159 tablet_copy_service.cc:161] P 3fe2aae583d14ad2a6892ae954d49708: Beginning new tablet copy session on tablet 8cafb41bf92a4f36877bf48362aee7a1 from peer 8f19408d91a24816b660335dfbd0685a at {username='slave'} at 127.0.0.1:37608: session id = 8f19408d91a24816b660335dfbd0685a-8cafb41bf92a4f36877bf48362aee7a1
I20240503 08:10:15.475190 2159 tablet_copy_source_session.cc:205] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:15.475499 2374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cafb41bf92a4f36877bf48362aee7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:15.476454 2374 tablet_copy_client.cc:783] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:15.476577 2374 tablet_copy_client.cc:647] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:15.477167 2374 tablet_copy_client.cc:523] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:15.477780 2374 tablet_bootstrap.cc:492] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: Bootstrap starting.
I20240503 08:10:15.478880 2374 tablet_bootstrap.cc:492] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:15.479015 2374 tablet_bootstrap.cc:492] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: Bootstrap complete.
I20240503 08:10:15.479084 2374 ts_tablet_manager.cc:1367] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: Time spent bootstrapping tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:10:15.479183 2374 raft_consensus.cc:348] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:15.479243 2374 raft_consensus.cc:729] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8f19408d91a24816b660335dfbd0685a, State: Initialized, Role: LEARNER
I20240503 08:10:15.479264 2374 consensus_queue.cc:260] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: true } }
I20240503 08:10:15.479344 2374 ts_tablet_manager.cc:1398] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:15.479562 2159 tablet_copy_service.cc:342] P 3fe2aae583d14ad2a6892ae954d49708: Request end of tablet copy session 8f19408d91a24816b660335dfbd0685a-8cafb41bf92a4f36877bf48362aee7a1 received from {username='slave'} at 127.0.0.1:37608
I20240503 08:10:15.479629 2159 tablet_copy_service.cc:433] P 3fe2aae583d14ad2a6892ae954d49708: ending tablet copy session 8f19408d91a24816b660335dfbd0685a-8cafb41bf92a4f36877bf48362aee7a1 on tablet 8cafb41bf92a4f36877bf48362aee7a1 with peer 8f19408d91a24816b660335dfbd0685a
I20240503 08:10:15.489167 2288 raft_consensus.cc:1203] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
W20240503 08:10:15.579738 1880 catalog_manager.cc:4573] TS 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1dd1d3f3edc54adbae4a580531a7b21b: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
W20240503 08:10:15.644887 1944 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 685c5510f4594a3e96c849a3c203118b not available for placement
I20240503 08:10:15.892870 2327 consensus_queue.cc:579] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Leader has been unable to successfully communicate with peer 685c5510f4594a3e96c849a3c203118b for more than 1 seconds (1.453s)
I20240503 08:10:15.971649 2385 raft_consensus.cc:1050] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3: attempting to promote NON_VOTER 8f19408d91a24816b660335dfbd0685a to VOTER
I20240503 08:10:15.971879 2385 consensus_queue.cc:237] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } }
I20240503 08:10:15.972389 2149 raft_consensus.cc:1261] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:15.972564 2288 raft_consensus.cc:1261] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [term 2 LEARNER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:15.972651 2386 consensus_queue.cc:1035] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:15.972761 2390 consensus_queue.cc:1035] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 08:10:15.972882 2038 consensus_peers.cc:485] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:15.973516 2386 raft_consensus.cc:2932] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:15.973657 2149 raft_consensus.cc:2932] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:15.974745 1894 catalog_manager.cc:5484] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 reported cstate change: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:10:15.974897 2288 raft_consensus.cc:2932] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:15.975853 2074 consensus_queue.cc:237] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } }
I20240503 08:10:15.976212 2149 raft_consensus.cc:1261] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:10:15.976372 2288 raft_consensus.cc:1261] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Refusing update from remote peer f3acedb5ad844e60b5f7a0b72383a6b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:10:15.976368 2327 consensus_queue.cc:1035] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:15.976527 2327 consensus_queue.cc:1035] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:15.976853 2385 raft_consensus.cc:2932] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:15.976958 2149 raft_consensus.cc:2932] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:15.977106 2288 raft_consensus.cc:2932] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:15.977943 1894 catalog_manager.cc:5484] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 reported cstate change: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New cstate: current_term: 2 leader_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:10:15.978183 1881 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4416cf3a68f546e49bf05e9e74b08773 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:16.002959 2288 raft_consensus.cc:1203] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 08:10:16.365087 2373 consensus_queue.cc:579] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Leader has been unable to successfully communicate with peer 685c5510f4594a3e96c849a3c203118b for more than 1 seconds (1.331s)
I20240503 08:10:16.563274 2399 raft_consensus.cc:1050] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708: attempting to promote NON_VOTER 8f19408d91a24816b660335dfbd0685a to VOTER
I20240503 08:10:16.563513 2399 consensus_queue.cc:237] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } }
I20240503 08:10:16.564067 2074 raft_consensus.cc:1261] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:16.564076 2288 raft_consensus.cc:1261] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [term 2 LEARNER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:16.564296 2399 consensus_queue.cc:1035] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:16.564421 2398 consensus_queue.cc:1035] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 08:10:16.564558 2113 consensus_peers.cc:485] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 -> Peer 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): Couldn't send request to peer 685c5510f4594a3e96c849a3c203118b. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:16.565191 2398 raft_consensus.cc:2932] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:16.565411 2288 raft_consensus.cc:2932] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:16.566308 2074 raft_consensus.cc:2932] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:16.566517 1893 catalog_manager.cc:5484] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 reported cstate change: config changed from index 2 to 3, 8f19408d91a24816b660335dfbd0685a (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3fe2aae583d14ad2a6892ae954d49708" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "685c5510f4594a3e96c849a3c203118b" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:10:16.567854 2149 consensus_queue.cc:237] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } }
I20240503 08:10:16.568169 2288 raft_consensus.cc:1261] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:10:16.568315 2074 raft_consensus.cc:1261] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Refusing update from remote peer 3fe2aae583d14ad2a6892ae954d49708: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:10:16.568351 2401 consensus_queue.cc:1035] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:16.568450 2399 consensus_queue.cc:1035] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:16.568848 2401 raft_consensus.cc:2932] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:16.569061 2074 raft_consensus.cc:2932] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:16.569847 2288 raft_consensus.cc:2932] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
I20240503 08:10:16.570659 1879 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cafb41bf92a4f36877bf48362aee7a1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:16.570708 1894 catalog_manager.cc:5484] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a reported cstate change: config changed from index 3 to 4, VOTER 685c5510f4594a3e96c849a3c203118b (127.28.55.65) evicted. New cstate: current_term: 2 leader_uuid: "3fe2aae583d14ad2a6892ae954d49708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f3acedb5ad844e60b5f7a0b72383a6b3" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 37521 } } peers { permanent_uuid: "3fe2aae583d14ad2a6892ae954d49708" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 45635 } } peers { permanent_uuid: "8f19408d91a24816b660335dfbd0685a" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 44611 } attrs { promote: false } } }
W20240503 08:10:16.621690 1880 catalog_manager.cc:4573] TS 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4416cf3a68f546e49bf05e9e74b08773: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
W20240503 08:10:17.742054 1880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 08:10:17.742492 1880 catalog_manager.cc:4573] TS 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cafb41bf92a4f36877bf48362aee7a1: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
W20240503 08:10:18.781500 1880 catalog_manager.cc:4573] TS 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cafb41bf92a4f36877bf48362aee7a1: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
I20240503 08:10:20.033277 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:10:20.036190 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:20.036340 28893 tablet_replica.cc:330] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3: stopping tablet replica
I20240503 08:10:20.036406 28893 raft_consensus.cc:2229] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:20.036484 28893 raft_consensus.cc:2258] T 5a662efda5fc4f92a84284a3ffb1ef4b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.036661 28893 tablet_replica.cc:330] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3: stopping tablet replica
I20240503 08:10:20.036695 28893 raft_consensus.cc:2229] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:20.036736 28893 raft_consensus.cc:2258] T 1dd1d3f3edc54adbae4a580531a7b21b P f3acedb5ad844e60b5f7a0b72383a6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.036816 28893 tablet_replica.cc:330] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3: stopping tablet replica
I20240503 08:10:20.036842 28893 raft_consensus.cc:2229] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:10:20.036880 28893 raft_consensus.cc:2258] T 4416cf3a68f546e49bf05e9e74b08773 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.036968 28893 tablet_replica.cc:330] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3: stopping tablet replica
I20240503 08:10:20.037000 28893 raft_consensus.cc:2229] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:20.037039 28893 raft_consensus.cc:2258] T 8cafb41bf92a4f36877bf48362aee7a1 P f3acedb5ad844e60b5f7a0b72383a6b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.048823 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:10:20.050105 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:10:20.053100 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:20.053246 28893 tablet_replica.cc:330] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708: stopping tablet replica
I20240503 08:10:20.053328 28893 raft_consensus.cc:2229] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:20.053392 28893 raft_consensus.cc:2258] T 1dd1d3f3edc54adbae4a580531a7b21b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.053542 28893 tablet_replica.cc:330] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708: stopping tablet replica
I20240503 08:10:20.053572 28893 raft_consensus.cc:2229] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:20.053607 28893 raft_consensus.cc:2258] T 4416cf3a68f546e49bf05e9e74b08773 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.053687 28893 tablet_replica.cc:330] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708: stopping tablet replica
I20240503 08:10:20.053712 28893 raft_consensus.cc:2229] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:20.053748 28893 raft_consensus.cc:2258] T 5a662efda5fc4f92a84284a3ffb1ef4b P 3fe2aae583d14ad2a6892ae954d49708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.053830 28893 tablet_replica.cc:330] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708: stopping tablet replica
I20240503 08:10:20.053862 28893 raft_consensus.cc:2229] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:10:20.053891 28893 raft_consensus.cc:2258] T 8cafb41bf92a4f36877bf48362aee7a1 P 3fe2aae583d14ad2a6892ae954d49708 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.065742 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:10:20.067010 28893 tablet_server.cc:178] TabletServer@127.28.55.68:0 shutting down...
I20240503 08:10:20.069458 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:20.069577 28893 tablet_replica.cc:330] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a: stopping tablet replica
I20240503 08:10:20.069635 28893 raft_consensus.cc:2229] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:20.069706 28893 raft_consensus.cc:2258] T 8cafb41bf92a4f36877bf48362aee7a1 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.069901 28893 tablet_replica.cc:330] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a: stopping tablet replica
I20240503 08:10:20.069945 28893 raft_consensus.cc:2229] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:20.069993 28893 raft_consensus.cc:2258] T 4416cf3a68f546e49bf05e9e74b08773 P 8f19408d91a24816b660335dfbd0685a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.070106 28893 tablet_replica.cc:330] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a: stopping tablet replica
I20240503 08:10:20.070139 28893 raft_consensus.cc:2229] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:20.070183 28893 raft_consensus.cc:2258] T 1dd1d3f3edc54adbae4a580531a7b21b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.070291 28893 tablet_replica.cc:330] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a: stopping tablet replica
I20240503 08:10:20.070322 28893 raft_consensus.cc:2229] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:20.070358 28893 raft_consensus.cc:2258] T 5a662efda5fc4f92a84284a3ffb1ef4b P 8f19408d91a24816b660335dfbd0685a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:20.081959 28893 tablet_server.cc:194] TabletServer@127.28.55.68:0 shutdown complete.
I20240503 08:10:20.083161 28893 master.cc:537] Master@127.28.55.126:45797 shutting down...
W20240503 08:10:20.282577 1880 catalog_manager.cc:4573] TS 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4416cf3a68f546e49bf05e9e74b08773: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
W20240503 08:10:22.849037 1880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240503 08:10:22.849468 1880 catalog_manager.cc:4573] TS 685c5510f4594a3e96c849a3c203118b (127.28.55.65:46523): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1dd1d3f3edc54adbae4a580531a7b21b: Network error: Client connection negotiation failed: client connection to 127.28.55.65:46523: connect: Connection refused (error 111)
I20240503 08:10:23.678751 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:23.678916 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:23.678938 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P a9862686ac8d4bb6883555c1bd6a77c6: stopping tablet replica
I20240503 08:10:23.691004 28893 master.cc:552] Master@127.28.55.126:45797 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13079 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20240503 08:10:23.697221 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:42333
I20240503 08:10:23.697393 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:23.700181 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:23.700161 2418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:23.700122 2419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:23.700130 2421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:23.700412 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:23.700457 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:23.700474 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723823700473 us; error 0 us; skew 500 ppm
I20240503 08:10:23.705191 28893 webserver.cc:460] Webserver started at http://127.28.55.126:37901/ using document root <none> and password file <none>
I20240503 08:10:23.705286 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:23.705335 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:23.705382 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:23.705667 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "2947e76de0574e968a77d84157569f0c"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:23.706590 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:23.707129 2426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.707274 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.707331 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "2947e76de0574e968a77d84157569f0c"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:23.707370 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:23.716867 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:23.717078 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:23.719681 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:42333
I20240503 08:10:23.721702 2478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:42333 every 8 connection(s)
I20240503 08:10:23.721875 2479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:23.723066 2479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c: Bootstrap starting.
I20240503 08:10:23.723335 2479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:23.723850 2479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c: No bootstrap required, opened a new log
I20240503 08:10:23.723963 2479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947e76de0574e968a77d84157569f0c" member_type: VOTER }
I20240503 08:10:23.724014 2479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:23.724026 2479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2947e76de0574e968a77d84157569f0c, State: Initialized, Role: FOLLOWER
I20240503 08:10:23.724061 2479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947e76de0574e968a77d84157569f0c" member_type: VOTER }
I20240503 08:10:23.724090 2479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:10:23.724107 2479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:10:23.724124 2479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:23.724630 2479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947e76de0574e968a77d84157569f0c" member_type: VOTER }
I20240503 08:10:23.724704 2479 leader_election.cc:302] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2947e76de0574e968a77d84157569f0c; no voters:
I20240503 08:10:23.724817 2479 leader_election.cc:288] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:10:23.724872 2482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:23.724973 2479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:10:23.725036 2482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 1 LEADER]: Becoming Leader. State: Replica: 2947e76de0574e968a77d84157569f0c, State: Running, Role: LEADER
I20240503 08:10:23.725106 2482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947e76de0574e968a77d84157569f0c" member_type: VOTER }
I20240503 08:10:23.725317 2483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2947e76de0574e968a77d84157569f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947e76de0574e968a77d84157569f0c" member_type: VOTER } }
I20240503 08:10:23.725325 2484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2947e76de0574e968a77d84157569f0c. Latest consensus state: current_term: 1 leader_uuid: "2947e76de0574e968a77d84157569f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2947e76de0574e968a77d84157569f0c" member_type: VOTER } }
I20240503 08:10:23.725406 2483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:23.725416 2484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:23.725577 2488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:10:23.725749 2488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:10:23.726323 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:10:23.726341 2488 catalog_manager.cc:1347] Generated new cluster ID: 9a18e0ce4f4d49d0956decb6b4284d1b
I20240503 08:10:23.726388 2488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:10:23.736086 2488 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:10:23.736516 2488 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:10:23.745133 2488 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c: Generated new TSK 0
I20240503 08:10:23.745282 2488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:10:23.758193 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:23.759536 2500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:23.759666 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:23.759714 2501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:23.759825 2503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:23.759929 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:23.759965 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:23.759981 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723823759982 us; error 0 us; skew 500 ppm
I20240503 08:10:23.760637 28893 webserver.cc:460] Webserver started at http://127.28.55.65:34509/ using document root <none> and password file <none>
I20240503 08:10:23.760725 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:23.760769 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:23.760829 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:23.761122 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "6e3fca115e0a430f8ea356a294cb5bff"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:23.761960 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:23.762405 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.762545 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.762584 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "6e3fca115e0a430f8ea356a294cb5bff"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:23.762629 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:23.773114 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:23.773325 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:23.773474 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:23.773679 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:23.773706 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.773736 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:23.773761 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.776485 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:43459
I20240503 08:10:23.776526 2571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:43459 every 8 connection(s)
I20240503 08:10:23.776859 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:23.778115 2576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:23.778170 2579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:23.778353 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:23.778363 2577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:23.778538 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:23.778570 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:23.778582 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723823778582 us; error 0 us; skew 500 ppm
I20240503 08:10:23.779270 28893 webserver.cc:460] Webserver started at http://127.28.55.66:45883/ using document root <none> and password file <none>
I20240503 08:10:23.779345 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:23.779388 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:23.779435 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:23.779695 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "af93b0f512054d2dabe0343d372cb06d"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:23.780612 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:23.780740 2572 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42333
I20240503 08:10:23.780808 2572 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:23.780911 2572 heartbeater.cc:511] Master 127.28.55.126:42333 requested a full tablet report, sending...
I20240503 08:10:23.781064 2584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.781170 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.781167 2443 ts_manager.cc:194] Registered new tserver with Master: 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459)
I20240503 08:10:23.781227 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "af93b0f512054d2dabe0343d372cb06d"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:23.781273 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:23.781832 2443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57288
I20240503 08:10:23.792589 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:23.792867 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:23.793025 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:23.793234 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:23.793262 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.793289 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:23.793313 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.795975 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:34793
I20240503 08:10:23.796041 2647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:34793 every 8 connection(s)
I20240503 08:10:23.796348 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:23.797544 2651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:23.797613 2652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:23.797739 2654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:23.797662 28893 server_base.cc:1034] running on GCE node
I20240503 08:10:23.797988 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:23.798020 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:23.798031 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723823798031 us; error 0 us; skew 500 ppm
I20240503 08:10:23.798868 28893 webserver.cc:460] Webserver started at http://127.28.55.67:35411/ using document root <none> and password file <none>
I20240503 08:10:23.798955 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:23.799002 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:23.799050 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:23.799329 2648 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42333
I20240503 08:10:23.799363 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:23.799393 2648 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:23.799502 2648 heartbeater.cc:511] Master 127.28.55.126:42333 requested a full tablet report, sending...
I20240503 08:10:23.799722 2443 ts_manager.cc:194] Registered new tserver with Master: af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793)
I20240503 08:10:23.800078 2443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57290
I20240503 08:10:23.800390 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:23.800823 2659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.800938 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.800978 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:23.801023 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:23.810184 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:23.810382 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:23.810532 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:23.810742 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:23.810770 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.810799 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:23.810846 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:23.814108 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:33529
I20240503 08:10:23.814234 2722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:33529 every 8 connection(s)
I20240503 08:10:23.818298 2723 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42333
I20240503 08:10:23.818361 2723 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:23.818461 2723 heartbeater.cc:511] Master 127.28.55.126:42333 requested a full tablet report, sending...
I20240503 08:10:23.818686 2443 ts_manager.cc:194] Registered new tserver with Master: 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:23.818835 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004490543s
I20240503 08:10:23.819108 2443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57296
I20240503 08:10:24.782289 2572 heartbeater.cc:503] Master 127.28.55.126:42333 was elected leader, sending a full tablet report...
I20240503 08:10:24.800702 2648 heartbeater.cc:503] Master 127.28.55.126:42333 was elected leader, sending a full tablet report...
I20240503 08:10:24.819622 2723 heartbeater.cc:503] Master 127.28.55.126:42333 was elected leader, sending a full tablet report...
I20240503 08:10:24.844167 28893 test_util.cc:268] Using random seed: 1081589122
I20240503 08:10:24.849536 2443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57304:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:10:24.849704 2443 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:10:24.854840 2612 tablet_service.cc:1467] Processing CreateTablet for tablet f683d6d14c8b4a5299c9a06852204f0b (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:24.854840 2611 tablet_service.cc:1467] Processing CreateTablet for tablet eee084abb1a149879aaa893387695640 (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:24.855119 2536 tablet_service.cc:1467] Processing CreateTablet for tablet f683d6d14c8b4a5299c9a06852204f0b (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:24.854844 2610 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0930eb09e140d78d95ffffd1f68cb8 (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:24.855198 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f683d6d14c8b4a5299c9a06852204f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.855016 2612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f683d6d14c8b4a5299c9a06852204f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.855405 2533 tablet_service.cc:1467] Processing CreateTablet for tablet ae4fb452fc484078936fc63361b5c21a (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:24.855485 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4fb452fc484078936fc63361b5c21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.856734 2743 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: Bootstrap starting.
I20240503 08:10:24.856796 2535 tablet_service.cc:1467] Processing CreateTablet for tablet eee084abb1a149879aaa893387695640 (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:24.856900 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee084abb1a149879aaa893387695640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.854876 2609 tablet_service.cc:1467] Processing CreateTablet for tablet ae4fb452fc484078936fc63361b5c21a (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:24.857029 2610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0930eb09e140d78d95ffffd1f68cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.857143 2743 tablet_bootstrap.cc:654] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.857160 2609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4fb452fc484078936fc63361b5c21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.858108 2687 tablet_service.cc:1467] Processing CreateTablet for tablet f683d6d14c8b4a5299c9a06852204f0b (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:24.858198 2687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f683d6d14c8b4a5299c9a06852204f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.858394 2744 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap starting.
I20240503 08:10:24.858770 2744 tablet_bootstrap.cc:654] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.858917 2684 tablet_service.cc:1467] Processing CreateTablet for tablet ae4fb452fc484078936fc63361b5c21a (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:24.858996 2684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4fb452fc484078936fc63361b5c21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.859445 2747 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap starting.
I20240503 08:10:24.859526 2744 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: No bootstrap required, opened a new log
I20240503 08:10:24.859508 2686 tablet_service.cc:1467] Processing CreateTablet for tablet eee084abb1a149879aaa893387695640 (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:24.859593 2685 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0930eb09e140d78d95ffffd1f68cb8 (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:24.859622 2686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee084abb1a149879aaa893387695640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.856752 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0930eb09e140d78d95ffffd1f68cb8 (DEFAULT_TABLE table=test-workload [id=18dc94e92df14de4b159be8f748013f9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:24.859743 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0930eb09e140d78d95ffffd1f68cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.860086 2747 tablet_bootstrap.cc:654] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.860579 2747 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: No bootstrap required, opened a new log
I20240503 08:10:24.860612 2747 ts_tablet_manager.cc:1367] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:24.860731 2747 raft_consensus.cc:348] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.860770 2685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0930eb09e140d78d95ffffd1f68cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.860788 2747 raft_consensus.cc:374] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.860849 2747 raft_consensus.cc:729] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.860890 2747 consensus_queue.cc:260] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.861071 2747 ts_tablet_manager.cc:1398] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:24.861099 2611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee084abb1a149879aaa893387695640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:24.861152 2747 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap starting.
I20240503 08:10:24.861514 2747 tablet_bootstrap.cc:654] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.860762 2744 ts_tablet_manager.cc:1367] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:24.862102 2747 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: No bootstrap required, opened a new log
I20240503 08:10:24.862073 2744 raft_consensus.cc:348] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.862133 2747 ts_tablet_manager.cc:1367] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:24.862135 2744 raft_consensus.cc:374] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.862183 2744 raft_consensus.cc:729] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.862231 2744 consensus_queue.cc:260] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.862262 2747 raft_consensus.cc:348] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.862361 2747 raft_consensus.cc:374] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.862387 2744 ts_tablet_manager.cc:1398] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:24.862413 2747 raft_consensus.cc:729] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.862716 2747 consensus_queue.cc:260] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.862756 2743 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: No bootstrap required, opened a new log
I20240503 08:10:24.862874 2743 ts_tablet_manager.cc:1367] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:10:24.863036 2743 raft_consensus.cc:348] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.863085 2743 raft_consensus.cc:374] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.863102 2743 raft_consensus.cc:729] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.863138 2743 consensus_queue.cc:260] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.863286 2743 ts_tablet_manager.cc:1398] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:24.863329 2743 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Bootstrap starting.
I20240503 08:10:24.863622 2743 tablet_bootstrap.cc:654] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.862574 2744 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap starting.
I20240503 08:10:24.864100 2743 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: No bootstrap required, opened a new log
I20240503 08:10:24.864145 2743 ts_tablet_manager.cc:1367] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:24.864272 2744 tablet_bootstrap.cc:654] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.864265 2743 raft_consensus.cc:348] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.864369 2743 raft_consensus.cc:374] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.864399 2743 raft_consensus.cc:729] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.864444 2743 consensus_queue.cc:260] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.864535 2743 ts_tablet_manager.cc:1398] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:24.864585 2743 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Bootstrap starting.
I20240503 08:10:24.864791 2744 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: No bootstrap required, opened a new log
I20240503 08:10:24.864831 2744 ts_tablet_manager.cc:1367] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:24.864951 2743 tablet_bootstrap.cc:654] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.864979 2747 ts_tablet_manager.cc:1398] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:10:24.864976 2744 raft_consensus.cc:348] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.865068 2744 raft_consensus.cc:374] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.865085 2744 raft_consensus.cc:729] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.865131 2744 consensus_queue.cc:260] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.865243 2744 ts_tablet_manager.cc:1398] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:24.865293 2744 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap starting.
I20240503 08:10:24.865360 2747 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap starting.
I20240503 08:10:24.865695 2744 tablet_bootstrap.cc:654] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.865715 2747 tablet_bootstrap.cc:654] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.865833 2743 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: No bootstrap required, opened a new log
I20240503 08:10:24.865883 2743 ts_tablet_manager.cc:1367] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:24.865998 2743 raft_consensus.cc:348] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.866041 2743 raft_consensus.cc:374] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.866066 2743 raft_consensus.cc:729] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.866122 2743 consensus_queue.cc:260] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.866228 2743 ts_tablet_manager.cc:1398] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:24.866283 2743 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: Bootstrap starting.
I20240503 08:10:24.866395 2747 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: No bootstrap required, opened a new log
I20240503 08:10:24.866436 2747 ts_tablet_manager.cc:1367] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:24.866482 2744 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: No bootstrap required, opened a new log
I20240503 08:10:24.866518 2744 ts_tablet_manager.cc:1367] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:24.866576 2747 raft_consensus.cc:348] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.866645 2747 raft_consensus.cc:374] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.866667 2747 raft_consensus.cc:729] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.866611 2743 tablet_bootstrap.cc:654] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.866744 2747 consensus_queue.cc:260] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.866853 2747 ts_tablet_manager.cc:1398] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:24.866909 2747 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap starting.
I20240503 08:10:24.867274 2747 tablet_bootstrap.cc:654] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.867302 2743 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: No bootstrap required, opened a new log
I20240503 08:10:24.867388 2743 ts_tablet_manager.cc:1367] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:24.867550 2743 raft_consensus.cc:348] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.866659 2744 raft_consensus.cc:348] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.867681 2743 raft_consensus.cc:374] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.867781 2743 raft_consensus.cc:729] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.867801 2747 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: No bootstrap required, opened a new log
I20240503 08:10:24.867841 2747 ts_tablet_manager.cc:1367] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:24.867854 2743 consensus_queue.cc:260] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.867951 2743 ts_tablet_manager.cc:1398] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:24.867966 2747 raft_consensus.cc:348] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.868041 2747 raft_consensus.cc:374] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.868068 2747 raft_consensus.cc:729] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.867733 2744 raft_consensus.cc:374] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.868098 2744 raft_consensus.cc:729] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.868111 2747 consensus_queue.cc:260] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.868186 2744 consensus_queue.cc:260] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.868216 2747 ts_tablet_manager.cc:1398] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:24.868278 2744 ts_tablet_manager.cc:1398] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:10:24.868343 2744 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap starting.
I20240503 08:10:24.868367 2752 raft_consensus.cc:482] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:24.868410 2752 raft_consensus.cc:504] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.868546 2752 leader_election.cc:288] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:24.868700 2744 tablet_bootstrap.cc:654] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:24.869653 2744 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: No bootstrap required, opened a new log
I20240503 08:10:24.869742 2744 ts_tablet_manager.cc:1367] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:24.869891 2744 raft_consensus.cc:348] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.869949 2744 raft_consensus.cc:374] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:24.869966 2744 raft_consensus.cc:729] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Initialized, Role: FOLLOWER
I20240503 08:10:24.870038 2744 consensus_queue.cc:260] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.870138 2744 ts_tablet_manager.cc:1398] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:24.871776 2546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff" is_pre_election: true
I20240503 08:10:24.871865 2546 raft_consensus.cc:2454] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 0.
I20240503 08:10:24.871935 2697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" is_pre_election: true
I20240503 08:10:24.871985 2585 leader_election.cc:302] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, af93b0f512054d2dabe0343d372cb06d; no voters:
I20240503 08:10:24.872013 2697 raft_consensus.cc:2454] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 0.
I20240503 08:10:24.872083 2752 raft_consensus.cc:2781] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:24.872124 2752 raft_consensus.cc:482] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:24.872151 2752 raft_consensus.cc:3037] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:24.872751 2752 raft_consensus.cc:504] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.872879 2752 leader_election.cc:288] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 1 election: Requested vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:24.873026 2697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
I20240503 08:10:24.873062 2546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff"
I20240503 08:10:24.873121 2697 raft_consensus.cc:3037] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:24.873139 2546 raft_consensus.cc:3037] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:24.873739 2546 raft_consensus.cc:2454] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 1.
I20240503 08:10:24.873795 2697 raft_consensus.cc:2454] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 1.
I20240503 08:10:24.873881 2585 leader_election.cc:302] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, af93b0f512054d2dabe0343d372cb06d; no voters:
I20240503 08:10:24.873977 2752 raft_consensus.cc:2781] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:24.874114 2752 raft_consensus.cc:686] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 LEADER]: Becoming Leader. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Running, Role: LEADER
I20240503 08:10:24.874164 2752 consensus_queue.cc:237] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.874719 2443 catalog_manager.cc:5484] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d reported cstate change: term changed from 0 to 1, leader changed from <none> to af93b0f512054d2dabe0343d372cb06d (127.28.55.66). New cstate: current_term: 1 leader_uuid: "af93b0f512054d2dabe0343d372cb06d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:24.917191 2751 raft_consensus.cc:482] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:24.917330 2751 raft_consensus.cc:504] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.917567 2751 leader_election.cc:288] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:24.920272 2697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" is_pre_election: true
I20240503 08:10:24.920297 2622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d" is_pre_election: true
I20240503 08:10:24.920375 2697 raft_consensus.cc:2454] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 0.
I20240503 08:10:24.920396 2622 raft_consensus.cc:2454] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 0.
I20240503 08:10:24.920519 2510 leader_election.cc:302] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, 76d3057d1be2494db3fc58fc3cca3bfa; no voters:
I20240503 08:10:24.920625 2751 raft_consensus.cc:2781] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:24.920670 2751 raft_consensus.cc:482] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:24.920682 2751 raft_consensus.cc:3037] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:24.921348 2751 raft_consensus.cc:504] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.921475 2751 leader_election.cc:288] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 1 election: Requested vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:24.921643 2697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
I20240503 08:10:24.921708 2697 raft_consensus.cc:3037] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:24.921643 2622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d"
I20240503 08:10:24.921737 2622 raft_consensus.cc:3037] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:24.922271 2697 raft_consensus.cc:2454] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 1.
I20240503 08:10:24.922385 2622 raft_consensus.cc:2454] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 1.
I20240503 08:10:24.922473 2510 leader_election.cc:302] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, 76d3057d1be2494db3fc58fc3cca3bfa; no voters:
I20240503 08:10:24.922626 2751 raft_consensus.cc:2781] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:24.922775 2751 raft_consensus.cc:686] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 LEADER]: Becoming Leader. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Running, Role: LEADER
I20240503 08:10:24.922860 2751 consensus_queue.cc:237] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.923408 2443 catalog_manager.cc:5484] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65). New cstate: current_term: 1 leader_uuid: "6e3fca115e0a430f8ea356a294cb5bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:24.928684 2752 raft_consensus.cc:482] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:24.928759 2752 raft_consensus.cc:504] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.928897 2752 leader_election.cc:288] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:24.929073 2697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0930eb09e140d78d95ffffd1f68cb8" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" is_pre_election: true
I20240503 08:10:24.929073 2546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0930eb09e140d78d95ffffd1f68cb8" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff" is_pre_election: true
I20240503 08:10:24.929168 2697 raft_consensus.cc:2454] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 0.
I20240503 08:10:24.929170 2546 raft_consensus.cc:2454] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 0.
I20240503 08:10:24.929332 2586 leader_election.cc:302] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d3057d1be2494db3fc58fc3cca3bfa, af93b0f512054d2dabe0343d372cb06d; no voters:
I20240503 08:10:24.929441 2752 raft_consensus.cc:2781] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:24.929476 2752 raft_consensus.cc:482] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:24.929489 2752 raft_consensus.cc:3037] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:24.930099 2752 raft_consensus.cc:504] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.930217 2752 leader_election.cc:288] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 1 election: Requested vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:24.930331 2546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0930eb09e140d78d95ffffd1f68cb8" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff"
I20240503 08:10:24.930331 2697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0930eb09e140d78d95ffffd1f68cb8" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
I20240503 08:10:24.930416 2546 raft_consensus.cc:3037] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:24.930423 2697 raft_consensus.cc:3037] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:24.931123 2697 raft_consensus.cc:2454] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 1.
I20240503 08:10:24.931135 2546 raft_consensus.cc:2454] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 1.
I20240503 08:10:24.931308 2586 leader_election.cc:302] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, 76d3057d1be2494db3fc58fc3cca3bfa, af93b0f512054d2dabe0343d372cb06d; no voters:
I20240503 08:10:24.931412 2752 raft_consensus.cc:2781] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:24.931454 2752 raft_consensus.cc:686] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 LEADER]: Becoming Leader. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Running, Role: LEADER
I20240503 08:10:24.931507 2752 consensus_queue.cc:237] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:24.932085 2443 catalog_manager.cc:5484] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d reported cstate change: term changed from 0 to 1, leader changed from <none> to af93b0f512054d2dabe0343d372cb06d (127.28.55.66). New cstate: current_term: 1 leader_uuid: "af93b0f512054d2dabe0343d372cb06d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
W20240503 08:10:25.027784 2573 tablet.cc:2369] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:10:25.171224 2751 raft_consensus.cc:482] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:25.171365 2751 raft_consensus.cc:504] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:25.171546 2751 leader_election.cc:288] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:25.171785 2697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683d6d14c8b4a5299c9a06852204f0b" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" is_pre_election: true
I20240503 08:10:25.171792 2622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683d6d14c8b4a5299c9a06852204f0b" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d" is_pre_election: true
I20240503 08:10:25.171881 2697 raft_consensus.cc:2454] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 0.
I20240503 08:10:25.171900 2622 raft_consensus.cc:2454] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 0.
I20240503 08:10:25.172077 2509 leader_election.cc:302] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, af93b0f512054d2dabe0343d372cb06d; no voters:
I20240503 08:10:25.172190 2751 raft_consensus.cc:2781] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:25.172240 2751 raft_consensus.cc:482] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:25.172259 2751 raft_consensus.cc:3037] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:25.172957 2751 raft_consensus.cc:504] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:25.173095 2751 leader_election.cc:288] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 1 election: Requested vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:25.173300 2622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683d6d14c8b4a5299c9a06852204f0b" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d"
I20240503 08:10:25.173307 2697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683d6d14c8b4a5299c9a06852204f0b" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
I20240503 08:10:25.173363 2622 raft_consensus.cc:3037] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:25.173400 2697 raft_consensus.cc:3037] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:25.173972 2622 raft_consensus.cc:2454] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 1.
I20240503 08:10:25.173980 2697 raft_consensus.cc:2454] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 1.
I20240503 08:10:25.174152 2510 leader_election.cc:302] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, 76d3057d1be2494db3fc58fc3cca3bfa; no voters:
I20240503 08:10:25.174242 2751 raft_consensus.cc:2781] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:25.174286 2751 raft_consensus.cc:686] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 LEADER]: Becoming Leader. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Running, Role: LEADER
I20240503 08:10:25.174340 2751 consensus_queue.cc:237] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:25.174991 2443 catalog_manager.cc:5484] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65). New cstate: current_term: 1 leader_uuid: "6e3fca115e0a430f8ea356a294cb5bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:25.203564 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:10:25.206300 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:25.206493 28893 tablet_replica.cc:330] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: stopping tablet replica
I20240503 08:10:25.206563 28893 raft_consensus.cc:2229] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:25.206683 28893 pending_rounds.cc:62] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Trying to abort 1 pending ops.
I20240503 08:10:25.206720 28893 pending_rounds.cc:68] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508786884628480 op_type: NO_OP noop_request { }
I20240503 08:10:25.206769 28893 raft_consensus.cc:2866] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:10:25.206787 28893 raft_consensus.cc:2258] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.208722 28893 tablet_replica.cc:330] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: stopping tablet replica
I20240503 08:10:25.208783 28893 raft_consensus.cc:2229] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:25.208813 28893 raft_consensus.cc:2258] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.208981 28893 tablet_replica.cc:330] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: stopping tablet replica
I20240503 08:10:25.209017 28893 raft_consensus.cc:2229] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:25.209036 28893 raft_consensus.cc:2258] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.209185 28893 tablet_replica.cc:330] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: stopping tablet replica
I20240503 08:10:25.209225 28893 raft_consensus.cc:2229] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:25.209270 28893 pending_rounds.cc:62] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Trying to abort 1 pending ops.
I20240503 08:10:25.209292 28893 pending_rounds.cc:68] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508787914575872 op_type: NO_OP noop_request { }
I20240503 08:10:25.209354 28893 raft_consensus.cc:2866] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:10:25.209370 28893 raft_consensus.cc:2258] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.221632 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:10:25.222625 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:10:25.225378 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:25.225564 28893 tablet_replica.cc:330] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: stopping tablet replica
I20240503 08:10:25.225623 28893 raft_consensus.cc:2229] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:25.225673 28893 raft_consensus.cc:2258] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.225922 28893 tablet_replica.cc:330] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: stopping tablet replica
I20240503 08:10:25.225972 28893 raft_consensus.cc:2229] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:25.226023 28893 pending_rounds.cc:62] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Trying to abort 1 pending ops.
I20240503 08:10:25.226048 28893 pending_rounds.cc:68] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508786685030400 op_type: NO_OP noop_request { }
I20240503 08:10:25.226111 28893 raft_consensus.cc:2866] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:10:25.226147 28893 raft_consensus.cc:2258] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.226398 28893 tablet_replica.cc:330] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: stopping tablet replica
I20240503 08:10:25.226461 28893 raft_consensus.cc:2229] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:25.226500 28893 raft_consensus.cc:2258] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.226673 28893 tablet_replica.cc:330] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: stopping tablet replica
I20240503 08:10:25.226714 28893 raft_consensus.cc:2229] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:25.226753 28893 pending_rounds.cc:62] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Trying to abort 1 pending ops.
I20240503 08:10:25.226776 28893 pending_rounds.cc:68] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508786919948288 op_type: NO_OP noop_request { }
I20240503 08:10:25.226840 28893 raft_consensus.cc:2866] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:10:25.226868 28893 raft_consensus.cc:2258] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.228700 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:10:25.229612 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:10:25.232329 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:25.232472 28893 tablet_replica.cc:330] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: stopping tablet replica
I20240503 08:10:25.232538 28893 raft_consensus.cc:2229] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:25.232578 28893 raft_consensus.cc:2258] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.232770 28893 tablet_replica.cc:330] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: stopping tablet replica
I20240503 08:10:25.232810 28893 raft_consensus.cc:2229] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:25.232837 28893 raft_consensus.cc:2258] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.232991 28893 tablet_replica.cc:330] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: stopping tablet replica
I20240503 08:10:25.233027 28893 raft_consensus.cc:2229] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:25.233054 28893 raft_consensus.cc:2258] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.233263 28893 tablet_replica.cc:330] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: stopping tablet replica
I20240503 08:10:25.233299 28893 raft_consensus.cc:2229] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:25.233325 28893 raft_consensus.cc:2258] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:25.245151 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:10:25.246518 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:25.247722 2772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:25.247779 2771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:25.247908 2774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:25.248044 28893 server_base.cc:1034] running on GCE node
I20240503 08:10:25.248123 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:25.248142 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:25.248157 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723825248157 us; error 0 us; skew 500 ppm
I20240503 08:10:25.248811 28893 webserver.cc:460] Webserver started at http://127.28.55.68:33405/ using document root <none> and password file <none>
I20240503 08:10:25.248885 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:25.248931 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:25.248978 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:25.249259 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-3-root/instance:
uuid: "115689502b424a6baf84c69ed605611b"
format_stamp: "Formatted at 2024-05-03 08:10:25 on dist-test-slave-mlgg"
I20240503 08:10:25.250087 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:25.250542 2779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:25.250666 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:25.250715 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-3-root
uuid: "115689502b424a6baf84c69ed605611b"
format_stamp: "Formatted at 2024-05-03 08:10:25 on dist-test-slave-mlgg"
I20240503 08:10:25.250766 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:25.284286 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:25.284499 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:25.284651 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:25.284868 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:25.284900 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:25.284927 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:25.284955 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:25.287684 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.68:42609
I20240503 08:10:25.289742 2842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.68:42609 every 8 connection(s)
I20240503 08:10:25.292694 2843 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42333
I20240503 08:10:25.292760 2843 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:25.292845 2843 heartbeater.cc:511] Master 127.28.55.126:42333 requested a full tablet report, sending...
I20240503 08:10:25.293080 2443 ts_manager.cc:194] Registered new tserver with Master: 115689502b424a6baf84c69ed605611b (127.28.55.68:42609)
I20240503 08:10:25.293443 2443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57320
W20240503 08:10:25.727000 2491 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:25.727792 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:25.727854 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:25.728359 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:25.728421 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:25.728852 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:25.728894 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
I20240503 08:10:26.293965 2843 heartbeater.cc:503] Master 127.28.55.126:42333 was elected leader, sending a full tablet report...
W20240503 08:10:26.729830 2491 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:26.730613 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:26.730661 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:26.731324 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:26.731370 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:26.731827 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:26.731870 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:27.732894 2491 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:27.733781 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:27.733830 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:27.734339 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:27.734403 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:27.734786 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:27.734855 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:28.735874 2491 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:28.736397 2427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:10:28.736821 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:28.736876 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:28.737433 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:28.737463 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:28.737869 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:28.737917 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:29.739002 2491 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:29.739816 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:29.739858 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.65:43459: connect: Connection refused (error 111)
W20240503 08:10:29.740265 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:29.740293 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:29.740653 2491 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:29.740684 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.28.55.66:34793: connect: Connection refused (error 111)
W20240503 08:10:30.740993 2491 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6e3fca115e0a430f8ea356a294cb5bff not available for placement
I20240503 08:10:30.801786 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:30.803189 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:30.803282 2861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:30.803292 2864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:30.803169 2862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:30.803613 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:30.803645 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:30.803656 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723830803657 us; error 0 us; skew 500 ppm
I20240503 08:10:30.809952 28893 webserver.cc:460] Webserver started at http://127.28.55.65:34509/ using document root <none> and password file <none>
I20240503 08:10:30.810071 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:30.810106 28893 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:10:30.810652 28893 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.811297 2869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.811419 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.811463 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "6e3fca115e0a430f8ea356a294cb5bff"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:30.811522 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:30.817124 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:30.817327 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:30.817453 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:30.817752 2876 ts_tablet_manager.cc:516] Loading tablet metadata (0/4 complete)
I20240503 08:10:30.819165 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 08:10:30.819212 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20240503 08:10:30.819236 28893 ts_tablet_manager.cc:574] Registering tablets (0/4 complete)
I20240503 08:10:30.819676 2876 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap starting.
I20240503 08:10:30.820451 28893 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 08:10:30.820499 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:30.820818 2876 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:30.821004 2876 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap complete.
I20240503 08:10:30.821123 2876 ts_tablet_manager.cc:1367] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:30.821265 2876 raft_consensus.cc:348] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.821360 2876 raft_consensus.cc:729] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.821421 2876 consensus_queue.cc:260] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.821583 2876 ts_tablet_manager.cc:1398] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:30.821651 2876 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap starting.
I20240503 08:10:30.823639 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:43459
I20240503 08:10:30.823671 2936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:43459 every 8 connection(s)
I20240503 08:10:30.823915 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:30.824616 2876 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:10:30.824848 2876 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap complete.
I20240503 08:10:30.824965 2876 ts_tablet_manager.cc:1367] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:10:30.825109 2876 raft_consensus.cc:348] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:30.825150 2944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:30.825189 2876 raft_consensus.cc:729] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.825238 2876 consensus_queue.cc:260] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.825338 2876 ts_tablet_manager.cc:1398] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:30.825403 2876 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap starting.
W20240503 08:10:30.825479 2942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:30.825682 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:30.825814 2941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:30.825980 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:30.826009 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:30.826025 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723830826024 us; error 0 us; skew 500 ppm
I20240503 08:10:30.826395 2876 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:30.826570 2876 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap complete.
I20240503 08:10:30.826676 2876 ts_tablet_manager.cc:1367] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:30.826781 2876 raft_consensus.cc:348] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.826887 2876 raft_consensus.cc:729] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.826885 28893 webserver.cc:460] Webserver started at http://127.28.55.66:45883/ using document root <none> and password file <none>
I20240503 08:10:30.826951 2876 consensus_queue.cc:260] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.826987 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:30.827014 28893 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:10:30.827049 2876 ts_tablet_manager.cc:1398] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.827097 2876 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap starting.
I20240503 08:10:30.827771 2937 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42333
I20240503 08:10:30.827859 28893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:30.827872 2937 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:30.828028 2937 heartbeater.cc:511] Master 127.28.55.126:42333 requested a full tablet report, sending...
I20240503 08:10:30.828074 2876 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:10:30.828276 2876 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Bootstrap complete.
I20240503 08:10:30.828352 2876 ts_tablet_manager.cc:1367] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:30.828367 2443 ts_manager.cc:194] Re-registered known tserver with Master: 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459)
I20240503 08:10:30.828431 2876 raft_consensus.cc:348] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.828485 2876 raft_consensus.cc:729] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.828519 2876 consensus_queue.cc:260] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.828580 2876 ts_tablet_manager.cc:1398] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.828835 2443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41990
I20240503 08:10:30.829157 2951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.829288 28893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:30.829308 2937 heartbeater.cc:503] Master 127.28.55.126:42333 was elected leader, sending a full tablet report...
I20240503 08:10:30.829331 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "af93b0f512054d2dabe0343d372cb06d"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:30.829375 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:30.838759 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:30.839027 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:30.839190 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:30.841429 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 08:10:30.841493 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20240503 08:10:30.841974 2958 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Bootstrap starting.
I20240503 08:10:30.842753 28893 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 08:10:30.842828 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:30.843353 2958 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:10:30.843569 2958 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Bootstrap complete.
I20240503 08:10:30.843655 2958 ts_tablet_manager.cc:1367] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:10:30.843786 2958 raft_consensus.cc:348] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.843858 2958 raft_consensus.cc:729] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.843900 2958 consensus_queue.cc:260] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.844192 2958 ts_tablet_manager.cc:1398] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:30.844281 2958 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: Bootstrap starting.
I20240503 08:10:30.845233 2958 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:30.845389 2958 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: Bootstrap complete.
I20240503 08:10:30.845500 2958 ts_tablet_manager.cc:1367] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:30.845649 2958 raft_consensus.cc:348] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.845702 2958 raft_consensus.cc:729] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.845754 2958 consensus_queue.cc:260] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.845844 2958 ts_tablet_manager.cc:1398] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.845892 2958 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Bootstrap starting.
I20240503 08:10:30.846498 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:34793
I20240503 08:10:30.846546 3018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:34793 every 8 connection(s)
I20240503 08:10:30.846864 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:30.847048 2958 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:10:30.847281 2958 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Bootstrap complete.
I20240503 08:10:30.847401 2958 ts_tablet_manager.cc:1367] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:10:30.847585 2958 raft_consensus.cc:348] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.847664 2958 raft_consensus.cc:729] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.847707 2958 consensus_queue.cc:260] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.847800 2958 ts_tablet_manager.cc:1398] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.847898 2958 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: Bootstrap starting.
W20240503 08:10:30.848220 3024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:30.848301 3026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:30.848459 3023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:30.848819 28893 server_base.cc:1034] running on GCE node
I20240503 08:10:30.848881 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:30.848901 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:30.848915 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723830848916 us; error 0 us; skew 500 ppm
I20240503 08:10:30.848930 2958 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:30.849072 2958 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: Bootstrap complete.
I20240503 08:10:30.849160 2958 ts_tablet_manager.cc:1367] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:30.849289 2958 raft_consensus.cc:348] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.849341 2958 raft_consensus.cc:729] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.849403 2958 consensus_queue.cc:260] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.849541 2958 ts_tablet_manager.cc:1398] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.849767 28893 webserver.cc:460] Webserver started at http://127.28.55.67:35411/ using document root <none> and password file <none>
I20240503 08:10:30.849836 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:30.849869 28893 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:10:30.850574 3020 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42333
I20240503 08:10:30.850629 3020 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:30.850651 28893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:30.850745 3020 heartbeater.cc:511] Master 127.28.55.126:42333 requested a full tablet report, sending...
I20240503 08:10:30.851095 2443 ts_manager.cc:194] Re-registered known tserver with Master: af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793)
I20240503 08:10:30.851286 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.851401 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.851444 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
format_stamp: "Formatted at 2024-05-03 08:10:23 on dist-test-slave-mlgg"
I20240503 08:10:30.851495 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:30.851634 2443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41996
I20240503 08:10:30.852025 3020 heartbeater.cc:503] Master 127.28.55.126:42333 was elected leader, sending a full tablet report...
I20240503 08:10:30.871081 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:30.871354 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:30.871544 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:30.873065 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240503 08:10:30.873111 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:30.873579 3039 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap starting.
I20240503 08:10:30.874032 28893 ts_tablet_manager.cc:590] Registered 4 tablets
I20240503 08:10:30.874068 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:30.874696 3039 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:30.874876 3039 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap complete.
I20240503 08:10:30.874991 3039 ts_tablet_manager.cc:1367] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:30.875164 3039 raft_consensus.cc:348] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.875254 3039 raft_consensus.cc:729] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.875346 3039 consensus_queue.cc:260] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.875542 3039 ts_tablet_manager.cc:1398] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:30.875604 3039 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap starting.
I20240503 08:10:30.876761 3039 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:30.876916 3039 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap complete.
I20240503 08:10:30.877015 3039 ts_tablet_manager.cc:1367] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:30.877189 3039 raft_consensus.cc:348] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.877247 3039 raft_consensus.cc:729] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.877291 3039 consensus_queue.cc:260] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.877485 3039 ts_tablet_manager.cc:1398] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:30.877568 3039 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap starting.
I20240503 08:10:30.877738 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:33529
I20240503 08:10:30.877830 3099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:33529 every 8 connection(s)
I20240503 08:10:30.878710 3039 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:30.878896 3039 tablet_bootstrap.cc:492] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap complete.
I20240503 08:10:30.879009 3039 ts_tablet_manager.cc:1367] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:30.879161 3039 raft_consensus.cc:348] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.879225 3039 raft_consensus.cc:729] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.879272 3039 consensus_queue.cc:260] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.879367 3039 ts_tablet_manager.cc:1398] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:30.879429 3039 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap starting.
I20240503 08:10:30.880273 3039 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:30.880434 3039 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Bootstrap complete.
I20240503 08:10:30.880550 3039 ts_tablet_manager.cc:1367] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:30.880707 3039 raft_consensus.cc:348] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.880766 3039 raft_consensus.cc:729] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Initialized, Role: FOLLOWER
I20240503 08:10:30.880815 3039 consensus_queue.cc:260] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:30.880931 3039 ts_tablet_manager.cc:1398] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:30.881227 3100 heartbeater.cc:347] Connected to a master server at 127.28.55.126:42333
I20240503 08:10:30.881296 3100 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:30.881400 3100 heartbeater.cc:511] Master 127.28.55.126:42333 requested a full tablet report, sending...
I20240503 08:10:30.881696 2443 ts_manager.cc:194] Re-registered known tserver with Master: 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:30.882076 2443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42010
I20240503 08:10:30.882455 3100 heartbeater.cc:503] Master 127.28.55.126:42333 was elected leader, sending a full tablet report...
W20240503 08:10:31.745218 2491 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Replica 6e3fca115e0a430f8ea356a294cb5bff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } }
W20240503 08:10:31.745697 2491 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet eee084abb1a149879aaa893387695640, TS af93b0f512054d2dabe0343d372cb06d -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:31.745754 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet eee084abb1a149879aaa893387695640, TS af93b0f512054d2dabe0343d372cb06d -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:31.748721 2491 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ae4fb452fc484078936fc63361b5c21a, TS 76d3057d1be2494db3fc58fc3cca3bfa -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:31.748790 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ae4fb452fc484078936fc63361b5c21a, TS 76d3057d1be2494db3fc58fc3cca3bfa -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:31.749163 2491 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9d0930eb09e140d78d95ffffd1f68cb8, TS 6e3fca115e0a430f8ea356a294cb5bff -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:31.749210 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 9d0930eb09e140d78d95ffffd1f68cb8, TS 6e3fca115e0a430f8ea356a294cb5bff -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.070449 3108 raft_consensus.cc:482] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:32.070569 3108 raft_consensus.cc:504] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.070832 3108 leader_election.cc:288] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.074388 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0930eb09e140d78d95ffffd1f68cb8" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff" is_pre_election: true
I20240503 08:10:32.074402 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0930eb09e140d78d95ffffd1f68cb8" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" is_pre_election: true
I20240503 08:10:32.074534 2911 raft_consensus.cc:2454] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 1.
I20240503 08:10:32.074568 3073 raft_consensus.cc:2454] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 1.
I20240503 08:10:32.074723 2952 leader_election.cc:302] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, af93b0f512054d2dabe0343d372cb06d; no voters:
I20240503 08:10:32.074834 3108 raft_consensus.cc:2781] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:10:32.074898 3108 raft_consensus.cc:482] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:32.074930 3108 raft_consensus.cc:3037] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.075621 3108 raft_consensus.cc:504] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.075767 3108 leader_election.cc:288] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 2 election: Requested vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.075930 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0930eb09e140d78d95ffffd1f68cb8" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff"
I20240503 08:10:32.075928 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0930eb09e140d78d95ffffd1f68cb8" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
I20240503 08:10:32.076035 2911 raft_consensus.cc:3037] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.076048 3073 raft_consensus.cc:3037] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.076692 3073 raft_consensus.cc:2454] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 2.
I20240503 08:10:32.076718 2911 raft_consensus.cc:2454] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 2.
I20240503 08:10:32.076867 2953 leader_election.cc:302] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d3057d1be2494db3fc58fc3cca3bfa, af93b0f512054d2dabe0343d372cb06d; no voters:
I20240503 08:10:32.076967 3108 raft_consensus.cc:2781] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:32.077092 3108 raft_consensus.cc:686] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 2 LEADER]: Becoming Leader. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Running, Role: LEADER
I20240503 08:10:32.077152 3108 consensus_queue.cc:237] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.077740 2443 catalog_manager.cc:5484] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "af93b0f512054d2dabe0343d372cb06d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:32.226625 3118 raft_consensus.cc:482] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:32.226752 3118 raft_consensus.cc:504] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.227012 3118 leader_election.cc:288] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793)
I20240503 08:10:32.229760 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff" is_pre_election: true
I20240503 08:10:32.229768 2993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d" is_pre_election: true
I20240503 08:10:32.229921 2911 raft_consensus.cc:2396] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76d3057d1be2494db3fc58fc3cca3bfa for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 08:10:32.229911 2993 raft_consensus.cc:2454] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d3057d1be2494db3fc58fc3cca3bfa in term 1.
I20240503 08:10:32.230115 3033 leader_election.cc:302] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d3057d1be2494db3fc58fc3cca3bfa, af93b0f512054d2dabe0343d372cb06d; no voters:
I20240503 08:10:32.230222 3118 raft_consensus.cc:2781] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:10:32.230281 3118 raft_consensus.cc:482] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:32.230297 3118 raft_consensus.cc:3037] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.231027 3118 raft_consensus.cc:504] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.231246 3118 leader_election.cc:288] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [CANDIDATE]: Term 2 election: Requested vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793)
I20240503 08:10:32.231356 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff"
I20240503 08:10:32.231376 2993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d"
I20240503 08:10:32.231420 2911 raft_consensus.cc:3037] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.231427 2993 raft_consensus.cc:3037] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.232024 2911 raft_consensus.cc:2396] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 76d3057d1be2494db3fc58fc3cca3bfa for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 08:10:32.232084 2993 raft_consensus.cc:2454] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d3057d1be2494db3fc58fc3cca3bfa in term 2.
I20240503 08:10:32.232223 3033 leader_election.cc:302] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76d3057d1be2494db3fc58fc3cca3bfa, af93b0f512054d2dabe0343d372cb06d; no voters: 6e3fca115e0a430f8ea356a294cb5bff
I20240503 08:10:32.232306 3118 raft_consensus.cc:2781] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:32.232559 3118 raft_consensus.cc:686] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 LEADER]: Becoming Leader. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Running, Role: LEADER
I20240503 08:10:32.232636 3118 consensus_queue.cc:237] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.233206 2443 catalog_manager.cc:5484] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa reported cstate change: term changed from 1 to 2, leader changed from 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65) to 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67). New cstate: current_term: 2 leader_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:32.235324 3123 raft_consensus.cc:482] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:32.235383 3123 raft_consensus.cc:504] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.235544 3123 leader_election.cc:288] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.238106 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" is_pre_election: true
I20240503 08:10:32.238164 2993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d" is_pre_election: true
I20240503 08:10:32.238191 3073 raft_consensus.cc:2454] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 1.
I20240503 08:10:32.238251 2993 raft_consensus.cc:2396] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e3fca115e0a430f8ea356a294cb5bff for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 08:10:32.238363 2871 leader_election.cc:302] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, 76d3057d1be2494db3fc58fc3cca3bfa; no voters:
I20240503 08:10:32.238484 3123 raft_consensus.cc:2781] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:10:32.238525 3123 raft_consensus.cc:482] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:32.238543 3123 raft_consensus.cc:3037] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.239262 3123 raft_consensus.cc:504] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.239391 3123 leader_election.cc:288] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 2 election: Requested vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.239567 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
I20240503 08:10:32.239565 2993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d"
I20240503 08:10:32.239652 3073 raft_consensus.cc:3037] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.239663 2993 raft_consensus.cc:3037] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.240213 2993 raft_consensus.cc:2396] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e3fca115e0a430f8ea356a294cb5bff for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 08:10:32.240267 3073 raft_consensus.cc:2454] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 2.
I20240503 08:10:32.240422 2871 leader_election.cc:302] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, 76d3057d1be2494db3fc58fc3cca3bfa; no voters:
I20240503 08:10:32.240538 3123 raft_consensus.cc:2781] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:32.240696 3123 raft_consensus.cc:686] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 2 LEADER]: Becoming Leader. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Running, Role: LEADER
I20240503 08:10:32.240749 3123 consensus_queue.cc:237] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.241328 2443 catalog_manager.cc:5484] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff reported cstate change: term changed from 1 to 2, leader changed from af93b0f512054d2dabe0343d372cb06d (127.28.55.66) to 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65). New cstate: current_term: 2 leader_uuid: "6e3fca115e0a430f8ea356a294cb5bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:32.275424 3123 raft_consensus.cc:482] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:32.275544 3123 raft_consensus.cc:504] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.275718 3123 leader_election.cc:288] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.275956 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" is_pre_election: true
I20240503 08:10:32.275960 2993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d" is_pre_election: true
I20240503 08:10:32.276072 2993 raft_consensus.cc:2454] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 2.
I20240503 08:10:32.276276 2870 leader_election.cc:302] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, af93b0f512054d2dabe0343d372cb06d; no voters: 76d3057d1be2494db3fc58fc3cca3bfa
I20240503 08:10:32.276386 3123 raft_consensus.cc:2781] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 08:10:32.276428 3123 raft_consensus.cc:482] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:32.276448 3123 raft_consensus.cc:3037] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:10:32.277137 3123 raft_consensus.cc:504] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.277268 3123 leader_election.cc:288] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 3 election: Requested vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.277451 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
I20240503 08:10:32.277479 2993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee084abb1a149879aaa893387695640" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d"
I20240503 08:10:32.277545 2993 raft_consensus.cc:3037] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:10:32.278131 2993 raft_consensus.cc:2454] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 3.
I20240503 08:10:32.278333 2870 leader_election.cc:302] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, af93b0f512054d2dabe0343d372cb06d; no voters: 76d3057d1be2494db3fc58fc3cca3bfa
I20240503 08:10:32.278437 3123 raft_consensus.cc:2781] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:10:32.278506 3123 raft_consensus.cc:686] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 3 LEADER]: Becoming Leader. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Running, Role: LEADER
I20240503 08:10:32.278566 3123 consensus_queue.cc:237] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.279186 2443 catalog_manager.cc:5484] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff reported cstate change: term changed from 2 to 3, leader changed from 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67) to 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65). New cstate: current_term: 3 leader_uuid: "6e3fca115e0a430f8ea356a294cb5bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:32.340349 3123 raft_consensus.cc:482] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:32.340490 3123 raft_consensus.cc:504] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.340703 3123 leader_election.cc:288] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.340941 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683d6d14c8b4a5299c9a06852204f0b" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" is_pre_election: true
I20240503 08:10:32.340946 2993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683d6d14c8b4a5299c9a06852204f0b" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d" is_pre_election: true
I20240503 08:10:32.341049 3073 raft_consensus.cc:2454] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 1.
I20240503 08:10:32.341105 2993 raft_consensus.cc:2454] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 1.
I20240503 08:10:32.341220 2871 leader_election.cc:302] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, 76d3057d1be2494db3fc58fc3cca3bfa; no voters:
I20240503 08:10:32.341300 3123 raft_consensus.cc:2781] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:10:32.341341 3123 raft_consensus.cc:482] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:32.341358 3123 raft_consensus.cc:3037] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.342052 3123 raft_consensus.cc:504] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.342188 3123 leader_election.cc:288] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 2 election: Requested vote from peers af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.342343 2993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683d6d14c8b4a5299c9a06852204f0b" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "af93b0f512054d2dabe0343d372cb06d"
I20240503 08:10:32.342375 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683d6d14c8b4a5299c9a06852204f0b" candidate_uuid: "6e3fca115e0a430f8ea356a294cb5bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
I20240503 08:10:32.342412 3073 raft_consensus.cc:3037] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.342412 2993 raft_consensus.cc:3037] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:32.343159 2993 raft_consensus.cc:2454] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 2.
I20240503 08:10:32.343159 3073 raft_consensus.cc:2454] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3fca115e0a430f8ea356a294cb5bff in term 2.
I20240503 08:10:32.343379 2871 leader_election.cc:302] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3fca115e0a430f8ea356a294cb5bff, 76d3057d1be2494db3fc58fc3cca3bfa; no voters:
I20240503 08:10:32.343498 3123 raft_consensus.cc:2781] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:32.343561 3123 raft_consensus.cc:686] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 2 LEADER]: Becoming Leader. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Running, Role: LEADER
I20240503 08:10:32.343614 3123 consensus_queue.cc:237] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.344213 2443 catalog_manager.cc:5484] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "6e3fca115e0a430f8ea356a294cb5bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:32.415150 3108 raft_consensus.cc:482] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:32.415268 3108 raft_consensus.cc:504] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.415462 3108 leader_election.cc:288] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.415742 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" is_pre_election: true
I20240503 08:10:32.415747 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff" is_pre_election: true
I20240503 08:10:32.415855 3073 raft_consensus.cc:2454] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 2.
I20240503 08:10:32.416025 2953 leader_election.cc:302] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76d3057d1be2494db3fc58fc3cca3bfa, af93b0f512054d2dabe0343d372cb06d; no voters: 6e3fca115e0a430f8ea356a294cb5bff
I20240503 08:10:32.416122 3108 raft_consensus.cc:2781] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 08:10:32.416167 3108 raft_consensus.cc:482] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:32.416189 3108 raft_consensus.cc:3037] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:10:32.416869 3108 raft_consensus.cc:504] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.417008 3108 leader_election.cc:288] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 3 election: Requested vote from peers 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459), 76d3057d1be2494db3fc58fc3cca3bfa (127.28.55.67:33529)
I20240503 08:10:32.417172 3073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "76d3057d1be2494db3fc58fc3cca3bfa"
I20240503 08:10:32.417172 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4fb452fc484078936fc63361b5c21a" candidate_uuid: "af93b0f512054d2dabe0343d372cb06d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e3fca115e0a430f8ea356a294cb5bff"
I20240503 08:10:32.417274 3073 raft_consensus.cc:3037] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:10:32.417889 3073 raft_consensus.cc:2454] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af93b0f512054d2dabe0343d372cb06d in term 3.
I20240503 08:10:32.418078 2953 leader_election.cc:302] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76d3057d1be2494db3fc58fc3cca3bfa, af93b0f512054d2dabe0343d372cb06d; no voters: 6e3fca115e0a430f8ea356a294cb5bff
I20240503 08:10:32.418164 3108 raft_consensus.cc:2781] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:10:32.418218 3108 raft_consensus.cc:686] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 3 LEADER]: Becoming Leader. State: Replica: af93b0f512054d2dabe0343d372cb06d, State: Running, Role: LEADER
I20240503 08:10:32.418270 3108 consensus_queue.cc:237] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.418865 2443 catalog_manager.cc:5484] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d reported cstate change: term changed from 2 to 3, leader changed from 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65) to af93b0f512054d2dabe0343d372cb06d (127.28.55.66). New cstate: current_term: 3 leader_uuid: "af93b0f512054d2dabe0343d372cb06d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:32.575206 3073 raft_consensus.cc:1261] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Refusing update from remote peer af93b0f512054d2dabe0343d372cb06d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:10:32.575479 3108 consensus_queue.cc:1035] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:32.577172 2911 raft_consensus.cc:1261] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Refusing update from remote peer af93b0f512054d2dabe0343d372cb06d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:10:32.577445 3133 consensus_queue.cc:1035] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:32.656028 2993 raft_consensus.cc:1226] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 3 LEADER]: Rejecting Update request from peer 6e3fca115e0a430f8ea356a294cb5bff for earlier term 2. Current term is 3. Ops: []
I20240503 08:10:32.656387 3126 consensus_queue.cc:1046] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:32.656515 3126 consensus_queue.cc:1225] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Peer af93b0f512054d2dabe0343d372cb06d log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 08:10:32.656553 3123 raft_consensus.cc:3032] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:10:32.656605 3123 raft_consensus.cc:729] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e3fca115e0a430f8ea356a294cb5bff, State: Running, Role: LEADER
I20240503 08:10:32.656667 3123 consensus_queue.cc:260] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.656752 3123 raft_consensus.cc:3037] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:10:32.702279 2993 raft_consensus.cc:1226] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Rejecting Update request from peer 76d3057d1be2494db3fc58fc3cca3bfa for earlier term 2. Current term is 3. Ops: []
I20240503 08:10:32.702620 3121 consensus_queue.cc:1046] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:32.702764 3121 raft_consensus.cc:3032] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:10:32.702797 3121 raft_consensus.cc:729] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 76d3057d1be2494db3fc58fc3cca3bfa, State: Running, Role: LEADER
I20240503 08:10:32.702893 3121 consensus_queue.cc:260] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.702970 3121 raft_consensus.cc:3037] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:10:32.750208 2491 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240503 08:10:32.750659 2491 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f683d6d14c8b4a5299c9a06852204f0b, TS af93b0f512054d2dabe0343d372cb06d -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:32.750706 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet f683d6d14c8b4a5299c9a06852204f0b, TS af93b0f512054d2dabe0343d372cb06d -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:32.751104 2491 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ae4fb452fc484078936fc63361b5c21a, TS 76d3057d1be2494db3fc58fc3cca3bfa -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:32.751144 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ae4fb452fc484078936fc63361b5c21a, TS 76d3057d1be2494db3fc58fc3cca3bfa -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:32.751446 2491 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9d0930eb09e140d78d95ffffd1f68cb8, TS 6e3fca115e0a430f8ea356a294cb5bff -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
W20240503 08:10:32.751477 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 9d0930eb09e140d78d95ffffd1f68cb8, TS 6e3fca115e0a430f8ea356a294cb5bff -> TS 115689502b424a6baf84c69ed605611b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } }
I20240503 08:10:32.799820 2993 raft_consensus.cc:1261] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Refusing update from remote peer 6e3fca115e0a430f8ea356a294cb5bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:10:32.800127 3123 consensus_queue.cc:1035] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:32.800956 3073 raft_consensus.cc:1261] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Refusing update from remote peer 6e3fca115e0a430f8ea356a294cb5bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:10:32.801152 3126 consensus_queue.cc:1035] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:32.834589 3073 raft_consensus.cc:1261] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: Refusing update from remote peer 6e3fca115e0a430f8ea356a294cb5bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 08:10:32.834893 3126 consensus_queue.cc:1035] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:32.835005 3126 consensus_queue.cc:1225] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Peer 76d3057d1be2494db3fc58fc3cca3bfa log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 08:10:32.835273 3073 pending_rounds.cc:76] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Aborting all ops after (but not including) 0
I20240503 08:10:32.835337 3073 pending_rounds.cc:97] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240503 08:10:32.835352 3073 raft_consensus.cc:2866] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:10:32.836095 2993 raft_consensus.cc:1261] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Refusing update from remote peer 6e3fca115e0a430f8ea356a294cb5bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 08:10:32.836485 3123 consensus_queue.cc:1035] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:32.844120 2911 raft_consensus.cc:1261] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 3 FOLLOWER]: Refusing update from remote peer af93b0f512054d2dabe0343d372cb06d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 08:10:32.844379 3133 consensus_queue.cc:1035] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:32.844491 3133 consensus_queue.cc:1225] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [LEADER]: Peer 6e3fca115e0a430f8ea356a294cb5bff log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240503 08:10:32.844748 2911 pending_rounds.cc:76] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Aborting all ops after (but not including) 0
I20240503 08:10:32.844784 2911 pending_rounds.cc:97] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240503 08:10:32.844799 2911 raft_consensus.cc:2866] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:10:32.845571 3073 raft_consensus.cc:1261] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: Refusing update from remote peer af93b0f512054d2dabe0343d372cb06d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 08:10:32.845743 3142 consensus_queue.cc:1035] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:33.752305 2911 consensus_queue.cc:237] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } attrs { replace: true } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } }
I20240503 08:10:33.752969 3073 raft_consensus.cc:1261] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: Refusing update from remote peer 6e3fca115e0a430f8ea356a294cb5bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 08:10:33.753017 2993 raft_consensus.cc:1261] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Refusing update from remote peer 6e3fca115e0a430f8ea356a294cb5bff: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 08:10:33.753242 3123 consensus_queue.cc:1035] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:33.753357 3138 consensus_queue.cc:1035] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:33.754643 3126 raft_consensus.cc:2932] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } attrs { replace: true } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
I20240503 08:10:33.754791 3073 raft_consensus.cc:2932] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } attrs { replace: true } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
I20240503 08:10:33.756093 2993 raft_consensus.cc:2932] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } attrs { replace: true } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
W20240503 08:10:33.756175 2872 consensus_peers.cc:485] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff -> Peer 115689502b424a6baf84c69ed605611b (127.28.55.68:42609): Couldn't send request to peer 115689502b424a6baf84c69ed605611b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eee084abb1a149879aaa893387695640. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:33.756104 2442 catalog_manager.cc:5484] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff reported cstate change: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New cstate: current_term: 3 leader_uuid: "6e3fca115e0a430f8ea356a294cb5bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:33.756254 2911 consensus_queue.cc:237] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } attrs { replace: true } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } }
I20240503 08:10:33.756661 2991 raft_consensus.cc:1261] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Refusing update from remote peer 6e3fca115e0a430f8ea356a294cb5bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:33.756662 3073 raft_consensus.cc:1261] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Refusing update from remote peer 6e3fca115e0a430f8ea356a294cb5bff: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240503 08:10:33.756850 2872 consensus_peers.cc:485] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff -> Peer 115689502b424a6baf84c69ed605611b (127.28.55.68:42609): Couldn't send request to peer 115689502b424a6baf84c69ed605611b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f683d6d14c8b4a5299c9a06852204f0b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:33.756866 3138 consensus_queue.cc:1035] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:33.756973 3126 consensus_queue.cc:1035] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:33.757396 3123 raft_consensus.cc:2932] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } attrs { replace: true } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
I20240503 08:10:33.757479 3073 raft_consensus.cc:2932] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } attrs { replace: true } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
I20240503 08:10:33.757826 2993 raft_consensus.cc:2932] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } attrs { replace: true } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
I20240503 08:10:33.759159 2993 consensus_queue.cc:237] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } attrs { replace: true } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } }
I20240503 08:10:33.759303 2441 catalog_manager.cc:5484] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa reported cstate change: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New cstate: current_term: 2 leader_uuid: "6e3fca115e0a430f8ea356a294cb5bff" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } attrs { replace: true } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
I20240503 08:10:33.759650 2911 raft_consensus.cc:1261] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Refusing update from remote peer af93b0f512054d2dabe0343d372cb06d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:33.759856 3153 consensus_queue.cc:1035] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:33.759967 3073 raft_consensus.cc:1261] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Refusing update from remote peer af93b0f512054d2dabe0343d372cb06d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:33.760237 3112 consensus_queue.cc:1035] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:33.760766 2911 raft_consensus.cc:2932] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } attrs { replace: true } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
I20240503 08:10:33.760635 3153 raft_consensus.cc:2932] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } attrs { replace: true } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
I20240503 08:10:33.761821 2442 catalog_manager.cc:5484] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff reported cstate change: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New cstate: current_term: 2 leader_uuid: "af93b0f512054d2dabe0343d372cb06d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } attrs { replace: true } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
I20240503 08:10:33.762378 3073 raft_consensus.cc:2932] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 115689502b424a6baf84c69ed605611b (127.28.55.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } attrs { replace: true } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } } }
W20240503 08:10:33.763433 2954 consensus_peers.cc:485] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d -> Peer 115689502b424a6baf84c69ed605611b (127.28.55.68:42609): Couldn't send request to peer 115689502b424a6baf84c69ed605611b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d0930eb09e140d78d95ffffd1f68cb8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:34.214323 3157 ts_tablet_manager.cc:902] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: Initiating tablet copy from peer af93b0f512054d2dabe0343d372cb06d (127.28.55.66:34793)
I20240503 08:10:34.214699 3157 tablet_copy_client.cc:308] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:34793
I20240503 08:10:34.218340 3003 tablet_copy_service.cc:140] P af93b0f512054d2dabe0343d372cb06d: Received BeginTabletCopySession request for tablet 9d0930eb09e140d78d95ffffd1f68cb8 from peer 115689502b424a6baf84c69ed605611b ({username='slave'} at 127.0.0.1:38990)
I20240503 08:10:34.218425 3003 tablet_copy_service.cc:161] P af93b0f512054d2dabe0343d372cb06d: Beginning new tablet copy session on tablet 9d0930eb09e140d78d95ffffd1f68cb8 from peer 115689502b424a6baf84c69ed605611b at {username='slave'} at 127.0.0.1:38990: session id = 115689502b424a6baf84c69ed605611b-9d0930eb09e140d78d95ffffd1f68cb8
I20240503 08:10:34.219077 3003 tablet_copy_source_session.cc:205] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:34.219393 3157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0930eb09e140d78d95ffffd1f68cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:34.220872 3157 tablet_copy_client.cc:783] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:34.221101 3157 tablet_copy_client.cc:647] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:34.221786 3157 tablet_copy_client.cc:523] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:34.222698 3157 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: Bootstrap starting.
I20240503 08:10:34.224586 3157 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:34.224802 3157 tablet_bootstrap.cc:492] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: Bootstrap complete.
I20240503 08:10:34.224928 3157 ts_tablet_manager.cc:1367] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:10:34.225091 3157 raft_consensus.cc:348] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } attrs { replace: true } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } }
I20240503 08:10:34.225196 3157 raft_consensus.cc:729] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 115689502b424a6baf84c69ed605611b, State: Initialized, Role: LEARNER
I20240503 08:10:34.225250 3157 consensus_queue.cc:260] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } attrs { replace: true } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } }
I20240503 08:10:34.225464 3157 ts_tablet_manager.cc:1398] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:34.227013 3003 tablet_copy_service.cc:342] P af93b0f512054d2dabe0343d372cb06d: Request end of tablet copy session 115689502b424a6baf84c69ed605611b-9d0930eb09e140d78d95ffffd1f68cb8 received from {username='slave'} at 127.0.0.1:38990
I20240503 08:10:34.227069 3003 tablet_copy_service.cc:433] P af93b0f512054d2dabe0343d372cb06d: ending tablet copy session 115689502b424a6baf84c69ed605611b-9d0930eb09e140d78d95ffffd1f68cb8 on tablet 9d0930eb09e140d78d95ffffd1f68cb8 with peer 115689502b424a6baf84c69ed605611b
I20240503 08:10:34.237841 3157 ts_tablet_manager.cc:902] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: Initiating tablet copy from peer 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459)
I20240503 08:10:34.238180 3157 tablet_copy_client.cc:308] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.65:43459
I20240503 08:10:34.241494 2921 tablet_copy_service.cc:140] P 6e3fca115e0a430f8ea356a294cb5bff: Received BeginTabletCopySession request for tablet f683d6d14c8b4a5299c9a06852204f0b from peer 115689502b424a6baf84c69ed605611b ({username='slave'} at 127.0.0.1:38478)
I20240503 08:10:34.241559 2921 tablet_copy_service.cc:161] P 6e3fca115e0a430f8ea356a294cb5bff: Beginning new tablet copy session on tablet f683d6d14c8b4a5299c9a06852204f0b from peer 115689502b424a6baf84c69ed605611b at {username='slave'} at 127.0.0.1:38478: session id = 115689502b424a6baf84c69ed605611b-f683d6d14c8b4a5299c9a06852204f0b
I20240503 08:10:34.242039 2921 tablet_copy_source_session.cc:205] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:34.242372 3157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f683d6d14c8b4a5299c9a06852204f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:34.243655 3157 tablet_copy_client.cc:783] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:34.243772 3157 tablet_copy_client.cc:647] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:34.244278 3157 tablet_copy_client.cc:523] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:34.244997 3157 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: Bootstrap starting.
I20240503 08:10:34.246577 3157 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:34.246757 3157 tablet_bootstrap.cc:492] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: Bootstrap complete.
I20240503 08:10:34.246887 3157 ts_tablet_manager.cc:1367] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:10:34.247112 3157 raft_consensus.cc:348] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } attrs { replace: true } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } }
I20240503 08:10:34.247206 3157 raft_consensus.cc:729] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 115689502b424a6baf84c69ed605611b, State: Initialized, Role: LEARNER
I20240503 08:10:34.247238 3157 consensus_queue.cc:260] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } attrs { replace: true } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } }
I20240503 08:10:34.247367 3157 ts_tablet_manager.cc:1398] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:34.247535 2921 tablet_copy_service.cc:342] P 6e3fca115e0a430f8ea356a294cb5bff: Request end of tablet copy session 115689502b424a6baf84c69ed605611b-f683d6d14c8b4a5299c9a06852204f0b received from {username='slave'} at 127.0.0.1:38478
I20240503 08:10:34.247584 2921 tablet_copy_service.cc:433] P 6e3fca115e0a430f8ea356a294cb5bff: ending tablet copy session 115689502b424a6baf84c69ed605611b-f683d6d14c8b4a5299c9a06852204f0b on tablet f683d6d14c8b4a5299c9a06852204f0b with peer 115689502b424a6baf84c69ed605611b
I20240503 08:10:34.309223 3157 ts_tablet_manager.cc:902] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: Initiating tablet copy from peer 6e3fca115e0a430f8ea356a294cb5bff (127.28.55.65:43459)
I20240503 08:10:34.309547 3157 tablet_copy_client.cc:308] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.65:43459
I20240503 08:10:34.309724 2921 tablet_copy_service.cc:140] P 6e3fca115e0a430f8ea356a294cb5bff: Received BeginTabletCopySession request for tablet eee084abb1a149879aaa893387695640 from peer 115689502b424a6baf84c69ed605611b ({username='slave'} at 127.0.0.1:38478)
I20240503 08:10:34.309778 2921 tablet_copy_service.cc:161] P 6e3fca115e0a430f8ea356a294cb5bff: Beginning new tablet copy session on tablet eee084abb1a149879aaa893387695640 from peer 115689502b424a6baf84c69ed605611b at {username='slave'} at 127.0.0.1:38478: session id = 115689502b424a6baf84c69ed605611b-eee084abb1a149879aaa893387695640
I20240503 08:10:34.310334 2921 tablet_copy_source_session.cc:205] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:34.310694 3157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee084abb1a149879aaa893387695640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:34.312204 3157 tablet_copy_client.cc:783] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:34.312330 3157 tablet_copy_client.cc:647] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:34.312856 3157 tablet_copy_client.cc:523] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:34.313623 3157 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: Bootstrap starting.
I20240503 08:10:34.315155 3157 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:34.315338 3157 tablet_bootstrap.cc:492] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: Bootstrap complete.
I20240503 08:10:34.315436 3157 ts_tablet_manager.cc:1367] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:10:34.315608 3157 raft_consensus.cc:348] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } attrs { replace: true } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } }
I20240503 08:10:34.315698 3157 raft_consensus.cc:729] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 115689502b424a6baf84c69ed605611b, State: Initialized, Role: LEARNER
I20240503 08:10:34.315732 3157 consensus_queue.cc:260] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e3fca115e0a430f8ea356a294cb5bff" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 43459 } } peers { permanent_uuid: "af93b0f512054d2dabe0343d372cb06d" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 34793 } } peers { permanent_uuid: "76d3057d1be2494db3fc58fc3cca3bfa" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 33529 } attrs { replace: true } } peers { permanent_uuid: "115689502b424a6baf84c69ed605611b" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 42609 } attrs { promote: true } }
I20240503 08:10:34.315842 3157 ts_tablet_manager.cc:1398] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:34.316009 2921 tablet_copy_service.cc:342] P 6e3fca115e0a430f8ea356a294cb5bff: Request end of tablet copy session 115689502b424a6baf84c69ed605611b-eee084abb1a149879aaa893387695640 received from {username='slave'} at 127.0.0.1:38478
I20240503 08:10:34.316064 2921 tablet_copy_service.cc:433] P 6e3fca115e0a430f8ea356a294cb5bff: ending tablet copy session 115689502b424a6baf84c69ed605611b-eee084abb1a149879aaa893387695640 on tablet eee084abb1a149879aaa893387695640 with peer 115689502b424a6baf84c69ed605611b
I20240503 08:10:34.624447 2817 raft_consensus.cc:1203] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 08:10:34.721724 2817 raft_consensus.cc:1203] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 08:10:34.785687 2817 raft_consensus.cc:1203] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240503 08:10:34.902909 28893 tablet_server.cc:178] TabletServer@127.28.55.65:43459 shutting down...
W20240503 08:10:34.903646 2486 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20240503 08:10:34.904146 2491 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:10:34.904218 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:10:34.904871 2491 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:10:34.904909 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 08:10:34.906306 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:34.906406 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.906488 28893 raft_consensus.cc:2229] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:10:34.906587 28893 raft_consensus.cc:2258] T eee084abb1a149879aaa893387695640 P 6e3fca115e0a430f8ea356a294cb5bff [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.906829 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.906883 28893 raft_consensus.cc:2229] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:34.906921 28893 raft_consensus.cc:2258] T ae4fb452fc484078936fc63361b5c21a P 6e3fca115e0a430f8ea356a294cb5bff [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.907058 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.907115 28893 raft_consensus.cc:2229] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:34.907148 28893 raft_consensus.cc:2258] T 9d0930eb09e140d78d95ffffd1f68cb8 P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.907253 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.907281 28893 raft_consensus.cc:2229] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:10:34.907339 28893 raft_consensus.cc:2258] T f683d6d14c8b4a5299c9a06852204f0b P 6e3fca115e0a430f8ea356a294cb5bff [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.919871 28893 tablet_server.cc:194] TabletServer@127.28.55.65:43459 shutdown complete.
I20240503 08:10:34.921382 28893 tablet_server.cc:178] TabletServer@127.28.55.66:34793 shutting down...
W20240503 08:10:34.921922 2491 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:10:34.921986 2491 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 08:10:34.924310 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:34.924407 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.924465 28893 raft_consensus.cc:2229] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:34.924518 28893 raft_consensus.cc:2258] T eee084abb1a149879aaa893387695640 P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.924667 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.924698 28893 raft_consensus.cc:2229] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:10:34.924739 28893 raft_consensus.cc:2258] T ae4fb452fc484078936fc63361b5c21a P af93b0f512054d2dabe0343d372cb06d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.924844 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.924886 28893 raft_consensus.cc:2229] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:10:34.924949 28893 raft_consensus.cc:2258] T 9d0930eb09e140d78d95ffffd1f68cb8 P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.925066 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.925103 28893 raft_consensus.cc:2229] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:34.925140 28893 raft_consensus.cc:2258] T f683d6d14c8b4a5299c9a06852204f0b P af93b0f512054d2dabe0343d372cb06d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.937011 28893 tablet_server.cc:194] TabletServer@127.28.55.66:34793 shutdown complete.
I20240503 08:10:34.938269 28893 tablet_server.cc:178] TabletServer@127.28.55.67:33529 shutting down...
I20240503 08:10:34.941138 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:34.941247 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.941308 28893 raft_consensus.cc:2229] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:34.941361 28893 raft_consensus.cc:2258] T eee084abb1a149879aaa893387695640 P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.941509 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.941548 28893 raft_consensus.cc:2229] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:34.941573 28893 raft_consensus.cc:2258] T ae4fb452fc484078936fc63361b5c21a P 76d3057d1be2494db3fc58fc3cca3bfa [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.941649 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.941676 28893 raft_consensus.cc:2229] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:34.941697 28893 raft_consensus.cc:2258] T 9d0930eb09e140d78d95ffffd1f68cb8 P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.941772 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:34.941803 28893 raft_consensus.cc:2229] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:34.941828 28893 raft_consensus.cc:2258] T f683d6d14c8b4a5299c9a06852204f0b P 76d3057d1be2494db3fc58fc3cca3bfa [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.953317 28893 tablet_server.cc:194] TabletServer@127.28.55.67:33529 shutdown complete.
I20240503 08:10:34.954438 28893 tablet_server.cc:178] TabletServer@127.28.55.68:0 shutting down...
I20240503 08:10:34.956743 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:34.956903 28893 tablet_replica.cc:330] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b: stopping tablet replica
I20240503 08:10:34.956969 28893 raft_consensus.cc:2229] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b [term 3 LEARNER]: Raft consensus shutting down.
I20240503 08:10:34.957023 28893 raft_consensus.cc:2258] T eee084abb1a149879aaa893387695640 P 115689502b424a6baf84c69ed605611b [term 3 LEARNER]: Raft consensus is shut down!
I20240503 08:10:34.957198 28893 tablet_replica.cc:330] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b: stopping tablet replica
I20240503 08:10:34.957230 28893 raft_consensus.cc:2229] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Raft consensus shutting down.
I20240503 08:10:34.957252 28893 raft_consensus.cc:2258] T 9d0930eb09e140d78d95ffffd1f68cb8 P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Raft consensus is shut down!
I20240503 08:10:34.957372 28893 tablet_replica.cc:330] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b: stopping tablet replica
I20240503 08:10:34.957401 28893 raft_consensus.cc:2229] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Raft consensus shutting down.
I20240503 08:10:34.957419 28893 raft_consensus.cc:2258] T f683d6d14c8b4a5299c9a06852204f0b P 115689502b424a6baf84c69ed605611b [term 2 LEARNER]: Raft consensus is shut down!
I20240503 08:10:34.969066 28893 tablet_server.cc:194] TabletServer@127.28.55.68:0 shutdown complete.
I20240503 08:10:34.970136 28893 master.cc:537] Master@127.28.55.126:42333 shutting down...
I20240503 08:10:34.972561 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:34.972638 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:34.972658 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2947e76de0574e968a77d84157569f0c: stopping tablet replica
I20240503 08:10:34.984290 28893 master.cc:552] Master@127.28.55.126:42333 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (11293 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20240503 08:10:34.990370 28893 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.55.126:35865
I20240503 08:10:34.990521 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:34.991739 3183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:34.991791 3180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:34.991739 3181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:34.992050 28893 server_base.cc:1034] running on GCE node
I20240503 08:10:34.992161 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:34.992189 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:34.992200 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723834992201 us; error 0 us; skew 500 ppm
I20240503 08:10:34.992802 28893 webserver.cc:460] Webserver started at http://127.28.55.126:40425/ using document root <none> and password file <none>
I20240503 08:10:34.992877 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:34.992923 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:34.992995 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:34.993283 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/master-0-root/instance:
uuid: "1f6309f3581d44c7b649b362a729990b"
format_stamp: "Formatted at 2024-05-03 08:10:34 on dist-test-slave-mlgg"
I20240503 08:10:34.994125 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:34.994539 3188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:34.994678 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:34.994724 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/master-0-root
uuid: "1f6309f3581d44c7b649b362a729990b"
format_stamp: "Formatted at 2024-05-03 08:10:34 on dist-test-slave-mlgg"
I20240503 08:10:34.994763 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:35.005810 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:35.006008 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:35.008494 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.126:35865
I20240503 08:10:35.011097 3240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.126:35865 every 8 connection(s)
I20240503 08:10:35.011240 3241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:35.012347 3241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b: Bootstrap starting.
I20240503 08:10:35.012596 3241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:35.013134 3241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b: No bootstrap required, opened a new log
I20240503 08:10:35.013240 3241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f6309f3581d44c7b649b362a729990b" member_type: VOTER }
I20240503 08:10:35.013291 3241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:35.013303 3241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6309f3581d44c7b649b362a729990b, State: Initialized, Role: FOLLOWER
I20240503 08:10:35.013337 3241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f6309f3581d44c7b649b362a729990b" member_type: VOTER }
I20240503 08:10:35.013367 3241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:10:35.013383 3241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:10:35.013399 3241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:35.013877 3241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f6309f3581d44c7b649b362a729990b" member_type: VOTER }
I20240503 08:10:35.013933 3241 leader_election.cc:302] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f6309f3581d44c7b649b362a729990b; no voters:
I20240503 08:10:35.014042 3241 leader_election.cc:288] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:10:35.014088 3244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:35.014189 3241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:10:35.014250 3244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 1 LEADER]: Becoming Leader. State: Replica: 1f6309f3581d44c7b649b362a729990b, State: Running, Role: LEADER
I20240503 08:10:35.014336 3244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f6309f3581d44c7b649b362a729990b" member_type: VOTER }
I20240503 08:10:35.014535 3245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f6309f3581d44c7b649b362a729990b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f6309f3581d44c7b649b362a729990b" member_type: VOTER } }
I20240503 08:10:35.014590 3245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:35.014717 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f6309f3581d44c7b649b362a729990b. Latest consensus state: current_term: 1 leader_uuid: "1f6309f3581d44c7b649b362a729990b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f6309f3581d44c7b649b362a729990b" member_type: VOTER } }
I20240503 08:10:35.014760 3250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:10:35.014773 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:35.014928 3250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:10:35.015523 3250 catalog_manager.cc:1347] Generated new cluster ID: ddef6d78ceb542d2a0ca444b1d650d3e
I20240503 08:10:35.015519 28893 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:10:35.015594 3250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:10:35.024087 3250 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:10:35.024577 3250 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:10:35.029942 3250 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b: Generated new TSK 0
I20240503 08:10:35.030053 3250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:10:35.031262 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:35.032532 3263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:35.032765 3262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:35.032740 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:35.032639 3265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:35.032990 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:35.033021 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:35.033032 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723835033033 us; error 0 us; skew 500 ppm
I20240503 08:10:35.033700 28893 webserver.cc:460] Webserver started at http://127.28.55.65:41825/ using document root <none> and password file <none>
I20240503 08:10:35.033769 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:35.033803 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:35.033833 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:35.034077 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root/instance:
uuid: "363ff42f68d84e7d95a885b2447673c2"
format_stamp: "Formatted at 2024-05-03 08:10:35 on dist-test-slave-mlgg"
I20240503 08:10:35.034899 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:35.035315 3270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.035449 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.035488 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "363ff42f68d84e7d95a885b2447673c2"
format_stamp: "Formatted at 2024-05-03 08:10:35 on dist-test-slave-mlgg"
I20240503 08:10:35.035529 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:35.041898 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:35.042104 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:35.042241 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:35.042438 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:35.042464 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.042486 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:35.042510 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.045248 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:37655
I20240503 08:10:35.045282 3333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:37655 every 8 connection(s)
I20240503 08:10:35.045648 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:35.046887 3338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:35.046983 3339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:35.047156 3341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:35.047263 28893 server_base.cc:1034] running on GCE node
I20240503 08:10:35.047348 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:35.047377 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:35.047401 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723835047401 us; error 0 us; skew 500 ppm
I20240503 08:10:35.048142 28893 webserver.cc:460] Webserver started at http://127.28.55.66:34923/ using document root <none> and password file <none>
I20240503 08:10:35.048234 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:35.048272 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:35.048313 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:35.048619 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root/instance:
uuid: "ad402dfffae4424eb09cef218f971013"
format_stamp: "Formatted at 2024-05-03 08:10:35 on dist-test-slave-mlgg"
I20240503 08:10:35.049281 3334 heartbeater.cc:347] Connected to a master server at 127.28.55.126:35865
I20240503 08:10:35.049345 3334 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:35.049443 3334 heartbeater.cc:511] Master 127.28.55.126:35865 requested a full tablet report, sending...
I20240503 08:10:35.049643 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:35.049733 3205 ts_manager.cc:194] Registered new tserver with Master: 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655)
I20240503 08:10:35.050128 3346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.050256 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.050302 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root
uuid: "ad402dfffae4424eb09cef218f971013"
format_stamp: "Formatted at 2024-05-03 08:10:35 on dist-test-slave-mlgg"
I20240503 08:10:35.050346 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:35.050494 3205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48508
I20240503 08:10:35.064566 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:35.064769 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:35.064971 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:35.065176 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:35.065205 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.065227 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:35.065250 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.068073 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.66:44797
I20240503 08:10:35.068132 3409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.66:44797 every 8 connection(s)
I20240503 08:10:35.068464 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:35.069658 3413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:35.069710 3414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:35.069839 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:35.069710 3416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:35.070268 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:35.070300 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:35.070312 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723835070312 us; error 0 us; skew 500 ppm
I20240503 08:10:35.071113 28893 webserver.cc:460] Webserver started at http://127.28.55.67:33427/ using document root <none> and password file <none>
I20240503 08:10:35.071208 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:35.071285 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:35.071354 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:35.071681 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root/instance:
uuid: "4dbefbfd77e7474a8a69b1c536fde51e"
format_stamp: "Formatted at 2024-05-03 08:10:35 on dist-test-slave-mlgg"
I20240503 08:10:35.072069 3410 heartbeater.cc:347] Connected to a master server at 127.28.55.126:35865
I20240503 08:10:35.072134 3410 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:35.072229 3410 heartbeater.cc:511] Master 127.28.55.126:35865 requested a full tablet report, sending...
I20240503 08:10:35.072449 3205 ts_manager.cc:194] Registered new tserver with Master: ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:35.072649 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:35.072876 3205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48524
I20240503 08:10:35.073164 3421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.073289 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.073328 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root
uuid: "4dbefbfd77e7474a8a69b1c536fde51e"
format_stamp: "Formatted at 2024-05-03 08:10:35 on dist-test-slave-mlgg"
I20240503 08:10:35.073375 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:35.098429 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:35.098665 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:35.098852 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:35.099067 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:35.099097 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.099126 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:35.099149 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:35.102059 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.67:38655
I20240503 08:10:35.102138 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.67:38655 every 8 connection(s)
I20240503 08:10:35.105300 3485 heartbeater.cc:347] Connected to a master server at 127.28.55.126:35865
I20240503 08:10:35.105367 3485 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:35.105458 3485 heartbeater.cc:511] Master 127.28.55.126:35865 requested a full tablet report, sending...
I20240503 08:10:35.105693 3205 ts_manager.cc:194] Registered new tserver with Master: 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:35.106022 3205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48532
I20240503 08:10:35.106667 28893 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004392409s
I20240503 08:10:36.051033 3334 heartbeater.cc:503] Master 127.28.55.126:35865 was elected leader, sending a full tablet report...
I20240503 08:10:36.073388 3410 heartbeater.cc:503] Master 127.28.55.126:35865 was elected leader, sending a full tablet report...
I20240503 08:10:36.106545 3485 heartbeater.cc:503] Master 127.28.55.126:35865 was elected leader, sending a full tablet report...
I20240503 08:10:36.132123 28893 test_util.cc:268] Using random seed: 1092877078
I20240503 08:10:36.141820 3205 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48540:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:10:36.142022 3205 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:10:36.147197 3298 tablet_service.cc:1467] Processing CreateTablet for tablet 85bb842493534b7cbc575a3fcf47a4c1 (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:36.147377 3298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bb842493534b7cbc575a3fcf47a4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.147558 3374 tablet_service.cc:1467] Processing CreateTablet for tablet 85bb842493534b7cbc575a3fcf47a4c1 (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:36.147644 3374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bb842493534b7cbc575a3fcf47a4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.148066 3449 tablet_service.cc:1467] Processing CreateTablet for tablet 85bb842493534b7cbc575a3fcf47a4c1 (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:36.148159 3449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bb842493534b7cbc575a3fcf47a4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.149063 3373 tablet_service.cc:1467] Processing CreateTablet for tablet 1a864e641f254544a457ddc37f4bba26 (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:36.149097 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb52224259f4909a1838a56ba8121a9 (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:36.149163 3296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb52224259f4909a1838a56ba8121a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.149205 3372 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb52224259f4909a1838a56ba8121a9 (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:36.149163 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a864e641f254544a457ddc37f4bba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.149353 3371 tablet_service.cc:1467] Processing CreateTablet for tablet 8b615b62f5f84fa387d8d07f6f2a1ade (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:36.149385 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb52224259f4909a1838a56ba8121a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.149483 3448 tablet_service.cc:1467] Processing CreateTablet for tablet 1a864e641f254544a457ddc37f4bba26 (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:36.149561 3448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a864e641f254544a457ddc37f4bba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.150552 3295 tablet_service.cc:1467] Processing CreateTablet for tablet 8b615b62f5f84fa387d8d07f6f2a1ade (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:36.150640 3295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b615b62f5f84fa387d8d07f6f2a1ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.150833 3447 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb52224259f4909a1838a56ba8121a9 (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:36.150907 3447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb52224259f4909a1838a56ba8121a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.149063 3297 tablet_service.cc:1467] Processing CreateTablet for tablet 1a864e641f254544a457ddc37f4bba26 (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:36.151791 3297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a864e641f254544a457ddc37f4bba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.151983 3446 tablet_service.cc:1467] Processing CreateTablet for tablet 8b615b62f5f84fa387d8d07f6f2a1ade (DEFAULT_TABLE table=test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:36.152060 3446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b615b62f5f84fa387d8d07f6f2a1ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.153098 3505 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013: Bootstrap starting.
I20240503 08:10:36.153129 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b615b62f5f84fa387d8d07f6f2a1ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.153198 3506 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:36.153573 3506 tablet_bootstrap.cc:654] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.154246 3505 tablet_bootstrap.cc:654] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.154644 3506 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: No bootstrap required, opened a new log
I20240503 08:10:36.154686 3506 ts_tablet_manager.cc:1367] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.154896 3507 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e: Bootstrap starting.
I20240503 08:10:36.154857 3506 raft_consensus.cc:348] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:36.155022 3506 raft_consensus.cc:374] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.155057 3505 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013: No bootstrap required, opened a new log
I20240503 08:10:36.155093 3506 raft_consensus.cc:729] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.155128 3505 ts_tablet_manager.cc:1367] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:36.155167 3506 consensus_queue.cc:260] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:36.155277 3507 tablet_bootstrap.cc:654] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.155360 3506 ts_tablet_manager.cc:1398] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.155429 3506 tablet_bootstrap.cc:492] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:36.155297 3505 raft_consensus.cc:348] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:36.155617 3505 raft_consensus.cc:374] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.155663 3505 raft_consensus.cc:729] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad402dfffae4424eb09cef218f971013, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.155727 3505 consensus_queue.cc:260] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:36.155881 3506 tablet_bootstrap.cc:654] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.155905 3505 ts_tablet_manager.cc:1398] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:36.156023 3505 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013: Bootstrap starting.
I20240503 08:10:36.156359 3505 tablet_bootstrap.cc:654] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.156591 3506 tablet_bootstrap.cc:492] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: No bootstrap required, opened a new log
I20240503 08:10:36.156625 3506 ts_tablet_manager.cc:1367] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.156733 3506 raft_consensus.cc:348] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.156896 3505 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013: No bootstrap required, opened a new log
I20240503 08:10:36.156960 3506 raft_consensus.cc:374] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.156965 3505 ts_tablet_manager.cc:1367] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.157033 3506 raft_consensus.cc:729] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.157092 3506 consensus_queue.cc:260] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.157186 3506 ts_tablet_manager.cc:1398] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:36.157174 3507 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e: No bootstrap required, opened a new log
I20240503 08:10:36.157241 3506 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:36.157264 3507 ts_tablet_manager.cc:1367] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:36.157174 3505 raft_consensus.cc:348] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.157331 3505 raft_consensus.cc:374] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.157351 3505 raft_consensus.cc:729] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad402dfffae4424eb09cef218f971013, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.157392 3505 consensus_queue.cc:260] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.157434 3507 raft_consensus.cc:348] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:36.157472 3505 ts_tablet_manager.cc:1398] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.157491 3507 raft_consensus.cc:374] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.157603 3506 tablet_bootstrap.cc:654] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.157534 3505 tablet_bootstrap.cc:492] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013: Bootstrap starting.
I20240503 08:10:36.157617 3507 raft_consensus.cc:729] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.157689 3507 consensus_queue.cc:260] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:36.157846 3507 ts_tablet_manager.cc:1398] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:36.157979 3505 tablet_bootstrap.cc:654] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.158123 3507 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e: Bootstrap starting.
I20240503 08:10:36.158479 3505 tablet_bootstrap.cc:492] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013: No bootstrap required, opened a new log
I20240503 08:10:36.158522 3505 ts_tablet_manager.cc:1367] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.158577 3507 tablet_bootstrap.cc:654] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.158646 3505 raft_consensus.cc:348] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.158699 3505 raft_consensus.cc:374] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.158711 3505 raft_consensus.cc:729] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad402dfffae4424eb09cef218f971013, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.158740 3505 consensus_queue.cc:260] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.158846 3505 ts_tablet_manager.cc:1398] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.158905 3505 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013: Bootstrap starting.
I20240503 08:10:36.159224 3505 tablet_bootstrap.cc:654] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.159402 3506 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: No bootstrap required, opened a new log
I20240503 08:10:36.159456 3506 ts_tablet_manager.cc:1367] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:36.159584 3506 raft_consensus.cc:348] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.159647 3506 raft_consensus.cc:374] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.159674 3506 raft_consensus.cc:729] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.159721 3506 consensus_queue.cc:260] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.159813 3506 ts_tablet_manager.cc:1398] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.159843 3505 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013: No bootstrap required, opened a new log
I20240503 08:10:36.159865 3515 raft_consensus.cc:482] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:36.159875 3505 ts_tablet_manager.cc:1367] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.159922 3515 raft_consensus.cc:504] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:36.159402 3507 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e: No bootstrap required, opened a new log
I20240503 08:10:36.160006 3507 ts_tablet_manager.cc:1367] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:10:36.160030 3505 raft_consensus.cc:348] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.160079 3505 raft_consensus.cc:374] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.159883 3506 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:36.160116 3507 raft_consensus.cc:348] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.160172 3507 raft_consensus.cc:374] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.160189 3507 raft_consensus.cc:729] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.160249 3507 consensus_queue.cc:260] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.160095 3505 raft_consensus.cc:729] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad402dfffae4424eb09cef218f971013, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.160506 3507 ts_tablet_manager.cc:1398] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.160599 3505 consensus_queue.cc:260] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.160085 3515 leader_election.cc:288] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.160688 3505 ts_tablet_manager.cc:1398] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:36.160517 3506 tablet_bootstrap.cc:654] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.160626 3507 tablet_bootstrap.cc:492] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e: Bootstrap starting.
I20240503 08:10:36.161458 3507 tablet_bootstrap.cc:654] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.161899 3506 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: No bootstrap required, opened a new log
I20240503 08:10:36.161952 3506 ts_tablet_manager.cc:1367] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:36.161978 3507 tablet_bootstrap.cc:492] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e: No bootstrap required, opened a new log
I20240503 08:10:36.162021 3507 ts_tablet_manager.cc:1367] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:36.162079 3506 raft_consensus.cc:348] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.162127 3506 raft_consensus.cc:374] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.162142 3506 raft_consensus.cc:729] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.162140 3507 raft_consensus.cc:348] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.162186 3507 raft_consensus.cc:374] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.162199 3507 raft_consensus.cc:729] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.162206 3506 consensus_queue.cc:260] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.162236 3507 consensus_queue.cc:260] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.162292 3506 ts_tablet_manager.cc:1398] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.162317 3507 ts_tablet_manager.cc:1398] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.162367 3507 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e: Bootstrap starting.
I20240503 08:10:36.162700 3507 tablet_bootstrap.cc:654] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.163378 3507 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e: No bootstrap required, opened a new log
I20240503 08:10:36.163427 3507 ts_tablet_manager.cc:1367] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.163709 3507 raft_consensus.cc:348] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.163760 3507 raft_consensus.cc:374] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.163776 3507 raft_consensus.cc:729] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.163811 3507 consensus_queue.cc:260] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.163901 3507 ts_tablet_manager.cc:1398] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.164079 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bb842493534b7cbc575a3fcf47a4c1" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2" is_pre_election: true
I20240503 08:10:36.164163 3308 raft_consensus.cc:2454] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 0.
I20240503 08:10:36.164356 3425 leader_election.cc:302] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, 4dbefbfd77e7474a8a69b1c536fde51e; no voters:
I20240503 08:10:36.164505 3521 raft_consensus.cc:2781] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:36.164541 3521 raft_consensus.cc:482] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:36.164553 3521 raft_consensus.cc:3037] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.165194 3521 raft_consensus.cc:504] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:36.165365 3521 leader_election.cc:288] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Requested vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.165457 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bb842493534b7cbc575a3fcf47a4c1" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2"
I20240503 08:10:36.165519 3308 raft_consensus.cc:3037] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.166208 3308 raft_consensus.cc:2454] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.166378 3425 leader_election.cc:302] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, 4dbefbfd77e7474a8a69b1c536fde51e; no voters:
I20240503 08:10:36.166601 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bb842493534b7cbc575a3fcf47a4c1" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013"
I20240503 08:10:36.166599 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85bb842493534b7cbc575a3fcf47a4c1" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013" is_pre_election: true
I20240503 08:10:36.166669 3383 raft_consensus.cc:3037] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.167305 3383 raft_consensus.cc:2454] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.167425 3521 raft_consensus.cc:2781] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:36.167485 3521 raft_consensus.cc:686] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Becoming Leader. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Running, Role: LEADER
I20240503 08:10:36.167527 3521 consensus_queue.cc:237] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:36.168052 3205 catalog_manager.cc:5484] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:36.186656 3510 raft_consensus.cc:482] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:36.186764 3510 raft_consensus.cc:504] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.187005 3510 leader_election.cc:288] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797), 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:36.190627 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b615b62f5f84fa387d8d07f6f2a1ade" candidate_uuid: "363ff42f68d84e7d95a885b2447673c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013" is_pre_election: true
I20240503 08:10:36.190773 3383 raft_consensus.cc:2454] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 363ff42f68d84e7d95a885b2447673c2 in term 0.
I20240503 08:10:36.190726 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b615b62f5f84fa387d8d07f6f2a1ade" candidate_uuid: "363ff42f68d84e7d95a885b2447673c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" is_pre_election: true
I20240503 08:10:36.190868 3459 raft_consensus.cc:2454] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 363ff42f68d84e7d95a885b2447673c2 in term 0.
I20240503 08:10:36.190984 3271 leader_election.cc:302] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:36.191073 3510 raft_consensus.cc:2781] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:36.191115 3510 raft_consensus.cc:482] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:36.191134 3510 raft_consensus.cc:3037] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.191906 3510 raft_consensus.cc:504] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.192039 3510 leader_election.cc:288] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [CANDIDATE]: Term 1 election: Requested vote from peers ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797), 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:36.192165 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b615b62f5f84fa387d8d07f6f2a1ade" candidate_uuid: "363ff42f68d84e7d95a885b2447673c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013"
I20240503 08:10:36.192224 3383 raft_consensus.cc:3037] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.192278 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b615b62f5f84fa387d8d07f6f2a1ade" candidate_uuid: "363ff42f68d84e7d95a885b2447673c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbefbfd77e7474a8a69b1c536fde51e"
I20240503 08:10:36.192355 3459 raft_consensus.cc:3037] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.192806 3383 raft_consensus.cc:2454] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 363ff42f68d84e7d95a885b2447673c2 in term 1.
I20240503 08:10:36.193001 3459 raft_consensus.cc:2454] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 363ff42f68d84e7d95a885b2447673c2 in term 1.
I20240503 08:10:36.193006 3271 leader_election.cc:302] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:36.193110 3510 raft_consensus.cc:2781] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:36.193256 3510 raft_consensus.cc:686] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 LEADER]: Becoming Leader. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Running, Role: LEADER
I20240503 08:10:36.193312 3510 consensus_queue.cc:237] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.193859 3205 catalog_manager.cc:5484] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65). New cstate: current_term: 1 leader_uuid: "363ff42f68d84e7d95a885b2447673c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:36.199092 3521 raft_consensus.cc:482] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:36.199177 3521 raft_consensus.cc:504] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.199303 3521 leader_election.cc:288] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.199498 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb52224259f4909a1838a56ba8121a9" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013" is_pre_election: true
I20240503 08:10:36.199498 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb52224259f4909a1838a56ba8121a9" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2" is_pre_election: true
I20240503 08:10:36.199610 3383 raft_consensus.cc:2454] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 0.
I20240503 08:10:36.199618 3308 raft_consensus.cc:2454] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 0.
I20240503 08:10:36.199764 3422 leader_election.cc:302] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbefbfd77e7474a8a69b1c536fde51e, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:36.199864 3521 raft_consensus.cc:2781] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:36.199898 3521 raft_consensus.cc:482] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:36.199924 3521 raft_consensus.cc:3037] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.200541 3521 raft_consensus.cc:504] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.200671 3521 leader_election.cc:288] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Requested vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.200824 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb52224259f4909a1838a56ba8121a9" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013"
I20240503 08:10:36.200806 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb52224259f4909a1838a56ba8121a9" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2"
I20240503 08:10:36.200901 3383 raft_consensus.cc:3037] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.200906 3308 raft_consensus.cc:3037] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.201556 3308 raft_consensus.cc:2454] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.201556 3383 raft_consensus.cc:2454] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.201758 3425 leader_election.cc:302] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, 4dbefbfd77e7474a8a69b1c536fde51e; no voters:
I20240503 08:10:36.201841 3521 raft_consensus.cc:2781] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:36.201900 3521 raft_consensus.cc:686] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Becoming Leader. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Running, Role: LEADER
I20240503 08:10:36.201953 3521 consensus_queue.cc:237] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.202535 3205 catalog_manager.cc:5484] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:36.343768 3521 raft_consensus.cc:482] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:36.343926 3521 raft_consensus.cc:504] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.344137 3521 leader_election.cc:288] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.344439 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a864e641f254544a457ddc37f4bba26" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2" is_pre_election: true
I20240503 08:10:36.344439 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a864e641f254544a457ddc37f4bba26" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013" is_pre_election: true
I20240503 08:10:36.344576 3308 raft_consensus.cc:2454] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 0.
I20240503 08:10:36.344578 3383 raft_consensus.cc:2454] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 0.
I20240503 08:10:36.344750 3425 leader_election.cc:302] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, 4dbefbfd77e7474a8a69b1c536fde51e; no voters:
I20240503 08:10:36.344874 3521 raft_consensus.cc:2781] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:36.344945 3521 raft_consensus.cc:482] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:36.344975 3521 raft_consensus.cc:3037] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.345644 3521 raft_consensus.cc:504] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.345775 3521 leader_election.cc:288] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Requested vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.345933 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a864e641f254544a457ddc37f4bba26" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2"
I20240503 08:10:36.345950 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a864e641f254544a457ddc37f4bba26" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013"
I20240503 08:10:36.346014 3308 raft_consensus.cc:3037] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.346021 3383 raft_consensus.cc:3037] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.346537 3383 raft_consensus.cc:2454] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.346551 3308 raft_consensus.cc:2454] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.346726 3422 leader_election.cc:302] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbefbfd77e7474a8a69b1c536fde51e, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:36.346822 3521 raft_consensus.cc:2781] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:36.346886 3521 raft_consensus.cc:686] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Becoming Leader. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Running, Role: LEADER
I20240503 08:10:36.346936 3521 consensus_queue.cc:237] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.347535 3205 catalog_manager.cc:5484] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:36.371870 28893 test_util.cc:268] Using random seed: 1093116816
I20240503 08:10:36.377218 3205 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48542:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:10:36.377391 3205 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:10:36.378986 3447 tablet_service.cc:1467] Processing CreateTablet for tablet a5bfef61532441b7b07943a3797f9a5c (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:36.379017 3295 tablet_service.cc:1467] Processing CreateTablet for tablet a5bfef61532441b7b07943a3797f9a5c (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:36.379019 3372 tablet_service.cc:1467] Processing CreateTablet for tablet a5bfef61532441b7b07943a3797f9a5c (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 08:10:36.379130 3447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5bfef61532441b7b07943a3797f9a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.379148 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5bfef61532441b7b07943a3797f9a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.379201 3296 tablet_service.cc:1467] Processing CreateTablet for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:36.379302 3446 tablet_service.cc:1467] Processing CreateTablet for tablet fe08e57f721a4ffe96c6311e34195923 (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:36.379391 3446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe08e57f721a4ffe96c6311e34195923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.380340 3373 tablet_service.cc:1467] Processing CreateTablet for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:36.380432 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.380470 3298 tablet_service.cc:1467] Processing CreateTablet for tablet 341bb356ba2d41c2a6f0fd5c36260645 (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:36.380607 3448 tablet_service.cc:1467] Processing CreateTablet for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 08:10:36.380692 3448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.381835 3449 tablet_service.cc:1467] Processing CreateTablet for tablet 341bb356ba2d41c2a6f0fd5c36260645 (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:36.381909 3449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341bb356ba2d41c2a6f0fd5c36260645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.382644 3507 tablet_bootstrap.cc:492] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: Bootstrap starting.
I20240503 08:10:36.380466 3505 tablet_bootstrap.cc:492] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: Bootstrap starting.
I20240503 08:10:36.383036 3507 tablet_bootstrap.cc:654] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.383335 3505 tablet_bootstrap.cc:654] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.378981 3297 tablet_service.cc:1467] Processing CreateTablet for tablet fe08e57f721a4ffe96c6311e34195923 (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:36.383658 3507 tablet_bootstrap.cc:492] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: No bootstrap required, opened a new log
I20240503 08:10:36.383706 3507 ts_tablet_manager.cc:1367] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:36.378981 3371 tablet_service.cc:1467] Processing CreateTablet for tablet fe08e57f721a4ffe96c6311e34195923 (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 08:10:36.383801 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe08e57f721a4ffe96c6311e34195923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.383862 3507 raft_consensus.cc:348] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.383937 3507 raft_consensus.cc:374] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.383958 3507 raft_consensus.cc:729] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.384013 3507 consensus_queue.cc:260] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.384110 3507 ts_tablet_manager.cc:1398] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.384158 3507 tablet_bootstrap.cc:492] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: Bootstrap starting.
I20240503 08:10:36.384531 3507 tablet_bootstrap.cc:654] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.379148 3295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5bfef61532441b7b07943a3797f9a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.385114 3298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341bb356ba2d41c2a6f0fd5c36260645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.385186 3507 tablet_bootstrap.cc:492] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: No bootstrap required, opened a new log
I20240503 08:10:36.385221 3507 ts_tablet_manager.cc:1367] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:36.385327 3507 raft_consensus.cc:348] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.385365 3507 raft_consensus.cc:374] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.385376 3507 raft_consensus.cc:729] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.385416 3507 consensus_queue.cc:260] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.385506 3507 ts_tablet_manager.cc:1398] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.385571 3507 tablet_bootstrap.cc:492] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: Bootstrap starting.
I20240503 08:10:36.385946 3507 tablet_bootstrap.cc:654] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.386143 3296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.386188 3506 tablet_bootstrap.cc:492] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:36.386500 3505 tablet_bootstrap.cc:492] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: No bootstrap required, opened a new log
I20240503 08:10:36.386538 3505 ts_tablet_manager.cc:1367] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:10:36.386551 3506 tablet_bootstrap.cc:654] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.386672 3505 raft_consensus.cc:348] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.386727 3505 raft_consensus.cc:374] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.386745 3505 raft_consensus.cc:729] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad402dfffae4424eb09cef218f971013, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.386786 3505 consensus_queue.cc:260] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.386926 3505 ts_tablet_manager.cc:1398] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:36.386948 3371 tablet_service.cc:1467] Processing CreateTablet for tablet 341bb356ba2d41c2a6f0fd5c36260645 (DEFAULT_TABLE table=dugtrio [id=1765e08367a641ff919f78e7506340a5]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 08:10:36.386994 3505 tablet_bootstrap.cc:492] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: Bootstrap starting.
I20240503 08:10:36.387030 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341bb356ba2d41c2a6f0fd5c36260645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.387368 3297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe08e57f721a4ffe96c6311e34195923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:36.387442 3505 tablet_bootstrap.cc:654] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.388144 3505 tablet_bootstrap.cc:492] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: No bootstrap required, opened a new log
I20240503 08:10:36.388183 3505 ts_tablet_manager.cc:1367] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.388314 3505 raft_consensus.cc:348] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.388319 3507 tablet_bootstrap.cc:492] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: No bootstrap required, opened a new log
I20240503 08:10:36.388438 3507 ts_tablet_manager.cc:1367] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:10:36.388577 3507 raft_consensus.cc:348] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.388626 3507 raft_consensus.cc:374] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.388664 3507 raft_consensus.cc:729] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.388629 3506 tablet_bootstrap.cc:492] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: No bootstrap required, opened a new log
I20240503 08:10:36.388698 3507 consensus_queue.cc:260] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.388732 3506 ts_tablet_manager.cc:1367] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:10:36.388782 3507 ts_tablet_manager.cc:1398] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.388846 3507 tablet_bootstrap.cc:492] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: Bootstrap starting.
I20240503 08:10:36.388871 3506 raft_consensus.cc:348] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.388926 3506 raft_consensus.cc:374] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.388943 3506 raft_consensus.cc:729] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.388993 3506 consensus_queue.cc:260] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.389103 3506 ts_tablet_manager.cc:1398] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.389163 3506 tablet_bootstrap.cc:492] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:36.389189 3507 tablet_bootstrap.cc:654] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.389498 3506 tablet_bootstrap.cc:654] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.389829 3507 tablet_bootstrap.cc:492] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: No bootstrap required, opened a new log
I20240503 08:10:36.389881 3507 ts_tablet_manager.cc:1367] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:36.390014 3507 raft_consensus.cc:348] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.390071 3507 raft_consensus.cc:374] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.390084 3507 raft_consensus.cc:729] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.390094 3506 tablet_bootstrap.cc:492] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: No bootstrap required, opened a new log
I20240503 08:10:36.390115 3507 consensus_queue.cc:260] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.390142 3506 ts_tablet_manager.cc:1367] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:36.390197 3507 ts_tablet_manager.cc:1398] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.390269 3506 raft_consensus.cc:348] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.390326 3506 raft_consensus.cc:374] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.390343 3506 raft_consensus.cc:729] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.390388 3506 consensus_queue.cc:260] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.388388 3505 raft_consensus.cc:374] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.390472 3505 raft_consensus.cc:729] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad402dfffae4424eb09cef218f971013, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.390476 3506 ts_tablet_manager.cc:1398] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.390550 3506 tablet_bootstrap.cc:492] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:36.390537 3505 consensus_queue.cc:260] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.390642 3505 ts_tablet_manager.cc:1398] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:36.390694 3505 tablet_bootstrap.cc:492] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: Bootstrap starting.
I20240503 08:10:36.390926 3506 tablet_bootstrap.cc:654] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.391060 3505 tablet_bootstrap.cc:654] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.391676 3505 tablet_bootstrap.cc:492] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: No bootstrap required, opened a new log
I20240503 08:10:36.391729 3505 ts_tablet_manager.cc:1367] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.391862 3505 raft_consensus.cc:348] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.391906 3505 raft_consensus.cc:374] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.391927 3505 raft_consensus.cc:729] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad402dfffae4424eb09cef218f971013, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.391970 3505 consensus_queue.cc:260] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.392066 3505 ts_tablet_manager.cc:1398] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.392122 3505 tablet_bootstrap.cc:492] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: Bootstrap starting.
I20240503 08:10:36.392470 3505 tablet_bootstrap.cc:654] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.393047 3505 tablet_bootstrap.cc:492] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: No bootstrap required, opened a new log
I20240503 08:10:36.393047 3506 tablet_bootstrap.cc:492] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: No bootstrap required, opened a new log
I20240503 08:10:36.393126 3505 ts_tablet_manager.cc:1367] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:36.393131 3506 ts_tablet_manager.cc:1367] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20240503 08:10:36.393298 3505 raft_consensus.cc:348] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.393298 3506 raft_consensus.cc:348] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.393381 3505 raft_consensus.cc:374] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.393388 3506 raft_consensus.cc:374] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.393417 3505 raft_consensus.cc:729] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad402dfffae4424eb09cef218f971013, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.393421 3506 raft_consensus.cc:729] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.393472 3505 consensus_queue.cc:260] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.393472 3506 consensus_queue.cc:260] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.393592 3505 ts_tablet_manager.cc:1398] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:36.393600 3506 ts_tablet_manager.cc:1398] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.393693 3506 tablet_bootstrap.cc:492] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:36.394081 3506 tablet_bootstrap.cc:654] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:36.394654 3506 tablet_bootstrap.cc:492] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: No bootstrap required, opened a new log
I20240503 08:10:36.394701 3506 ts_tablet_manager.cc:1367] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:36.394866 3506 raft_consensus.cc:348] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.394923 3506 raft_consensus.cc:374] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:36.394934 3506 raft_consensus.cc:729] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:36.394966 3506 consensus_queue.cc:260] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.395057 3506 ts_tablet_manager.cc:1398] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:36.400213 3512 raft_consensus.cc:482] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:36.400293 3512 raft_consensus.cc:504] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.400463 3512 leader_election.cc:288] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:36.404088 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cfe08358ea45a893b7c5dcc5ebeb4c" candidate_uuid: "ad402dfffae4424eb09cef218f971013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" is_pre_election: true
I20240503 08:10:36.404088 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cfe08358ea45a893b7c5dcc5ebeb4c" candidate_uuid: "ad402dfffae4424eb09cef218f971013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2" is_pre_election: true
I20240503 08:10:36.404232 3459 raft_consensus.cc:2454] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad402dfffae4424eb09cef218f971013 in term 0.
I20240503 08:10:36.404248 3308 raft_consensus.cc:2454] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad402dfffae4424eb09cef218f971013 in term 0.
I20240503 08:10:36.404400 3350 leader_election.cc:302] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbefbfd77e7474a8a69b1c536fde51e, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:36.404507 3512 raft_consensus.cc:2781] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:36.404556 3512 raft_consensus.cc:482] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:36.404577 3512 raft_consensus.cc:3037] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.405282 3512 raft_consensus.cc:504] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.405413 3512 leader_election.cc:288] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [CANDIDATE]: Term 1 election: Requested vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:36.405635 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cfe08358ea45a893b7c5dcc5ebeb4c" candidate_uuid: "ad402dfffae4424eb09cef218f971013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2"
I20240503 08:10:36.405702 3308 raft_consensus.cc:3037] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.405697 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cfe08358ea45a893b7c5dcc5ebeb4c" candidate_uuid: "ad402dfffae4424eb09cef218f971013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbefbfd77e7474a8a69b1c536fde51e"
I20240503 08:10:36.405790 3459 raft_consensus.cc:3037] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.406325 3308 raft_consensus.cc:2454] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad402dfffae4424eb09cef218f971013 in term 1.
I20240503 08:10:36.406370 3459 raft_consensus.cc:2454] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad402dfffae4424eb09cef218f971013 in term 1.
I20240503 08:10:36.406505 3350 leader_election.cc:302] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbefbfd77e7474a8a69b1c536fde51e, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:36.406605 3512 raft_consensus.cc:2781] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:36.406739 3512 raft_consensus.cc:686] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 1 LEADER]: Becoming Leader. State: Replica: ad402dfffae4424eb09cef218f971013, State: Running, Role: LEADER
I20240503 08:10:36.406790 3512 consensus_queue.cc:237] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.407419 3204 catalog_manager.cc:5484] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad402dfffae4424eb09cef218f971013 (127.28.55.66). New cstate: current_term: 1 leader_uuid: "ad402dfffae4424eb09cef218f971013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:36.428449 3521 raft_consensus.cc:482] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:36.428580 3521 raft_consensus.cc:504] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.428769 3521 leader_election.cc:288] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.428936 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341bb356ba2d41c2a6f0fd5c36260645" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2" is_pre_election: true
I20240503 08:10:36.429014 3308 raft_consensus.cc:2454] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 0.
I20240503 08:10:36.429016 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341bb356ba2d41c2a6f0fd5c36260645" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013" is_pre_election: true
I20240503 08:10:36.429143 3383 raft_consensus.cc:2454] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 0.
I20240503 08:10:36.429286 3425 leader_election.cc:302] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, 4dbefbfd77e7474a8a69b1c536fde51e, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:36.429414 3521 raft_consensus.cc:2781] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:36.429455 3521 raft_consensus.cc:482] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:36.429473 3521 raft_consensus.cc:3037] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.430181 3521 raft_consensus.cc:504] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.430317 3521 leader_election.cc:288] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Requested vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.430449 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341bb356ba2d41c2a6f0fd5c36260645" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013"
I20240503 08:10:36.430466 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341bb356ba2d41c2a6f0fd5c36260645" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2"
I20240503 08:10:36.430497 3383 raft_consensus.cc:3037] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.430521 3308 raft_consensus.cc:3037] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.430711 3510 raft_consensus.cc:482] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:36.430763 3510 raft_consensus.cc:504] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.430915 3510 leader_election.cc:288] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797), 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:36.431118 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe08e57f721a4ffe96c6311e34195923" candidate_uuid: "363ff42f68d84e7d95a885b2447673c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013" is_pre_election: true
I20240503 08:10:36.431128 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe08e57f721a4ffe96c6311e34195923" candidate_uuid: "363ff42f68d84e7d95a885b2447673c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" is_pre_election: true
I20240503 08:10:36.431198 3384 raft_consensus.cc:2454] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 363ff42f68d84e7d95a885b2447673c2 in term 0.
I20240503 08:10:36.431214 3459 raft_consensus.cc:2454] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 363ff42f68d84e7d95a885b2447673c2 in term 0.
I20240503 08:10:36.431253 3383 raft_consensus.cc:2454] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.431341 3308 raft_consensus.cc:2454] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.431380 3422 leader_election.cc:302] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbefbfd77e7474a8a69b1c536fde51e, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:36.431346 3271 leader_election.cc:302] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:36.431457 3521 raft_consensus.cc:2781] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:36.431495 3521 raft_consensus.cc:686] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Becoming Leader. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Running, Role: LEADER
I20240503 08:10:36.431541 3521 consensus_queue.cc:237] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.431576 3510 raft_consensus.cc:2781] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:36.431608 3510 raft_consensus.cc:482] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:36.431634 3510 raft_consensus.cc:3037] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.432121 3204 catalog_manager.cc:5484] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:36.432363 3510 raft_consensus.cc:504] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.432487 3510 leader_election.cc:288] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [CANDIDATE]: Term 1 election: Requested vote from peers ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797), 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:36.432616 3459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe08e57f721a4ffe96c6311e34195923" candidate_uuid: "363ff42f68d84e7d95a885b2447673c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbefbfd77e7474a8a69b1c536fde51e"
I20240503 08:10:36.432648 3383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe08e57f721a4ffe96c6311e34195923" candidate_uuid: "363ff42f68d84e7d95a885b2447673c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013"
I20240503 08:10:36.432677 3459 raft_consensus.cc:3037] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.432718 3383 raft_consensus.cc:3037] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.432893 3521 raft_consensus.cc:482] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:36.432935 3521 raft_consensus.cc:504] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.433033 3521 leader_election.cc:288] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.433117 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5bfef61532441b7b07943a3797f9a5c" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2" is_pre_election: true
I20240503 08:10:36.433174 3308 raft_consensus.cc:2454] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 0.
I20240503 08:10:36.433295 3425 leader_election.cc:302] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, 4dbefbfd77e7474a8a69b1c536fde51e; no voters:
I20240503 08:10:36.433368 3459 raft_consensus.cc:2454] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 363ff42f68d84e7d95a885b2447673c2 in term 1.
I20240503 08:10:36.433413 3521 raft_consensus.cc:2781] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:36.433342 3383 raft_consensus.cc:2454] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 363ff42f68d84e7d95a885b2447673c2 in term 1.
I20240503 08:10:36.433463 3521 raft_consensus.cc:482] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:36.433481 3521 raft_consensus.cc:3037] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.433521 3274 leader_election.cc:302] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, 4dbefbfd77e7474a8a69b1c536fde51e; no voters:
I20240503 08:10:36.433539 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5bfef61532441b7b07943a3797f9a5c" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013" is_pre_election: true
I20240503 08:10:36.433619 3510 raft_consensus.cc:2781] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:36.433625 3384 raft_consensus.cc:2454] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 0.
I20240503 08:10:36.433667 3510 raft_consensus.cc:686] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 LEADER]: Becoming Leader. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Running, Role: LEADER
I20240503 08:10:36.433706 3510 consensus_queue.cc:237] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.434064 3521 raft_consensus.cc:504] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.434180 3521 leader_election.cc:288] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Requested vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:36.434283 3384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5bfef61532441b7b07943a3797f9a5c" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad402dfffae4424eb09cef218f971013"
I20240503 08:10:36.434312 3308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5bfef61532441b7b07943a3797f9a5c" candidate_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2"
I20240503 08:10:36.434356 3384 raft_consensus.cc:3037] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.434360 3308 raft_consensus.cc:3037] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:36.434864 3308 raft_consensus.cc:2454] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.434952 3384 raft_consensus.cc:2454] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbefbfd77e7474a8a69b1c536fde51e in term 1.
I20240503 08:10:36.435083 3425 leader_election.cc:302] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ff42f68d84e7d95a885b2447673c2, 4dbefbfd77e7474a8a69b1c536fde51e; no voters:
I20240503 08:10:36.435165 3521 raft_consensus.cc:2781] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:36.435227 3521 raft_consensus.cc:686] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Becoming Leader. State: Replica: 4dbefbfd77e7474a8a69b1c536fde51e, State: Running, Role: LEADER
I20240503 08:10:36.435163 3204 catalog_manager.cc:5484] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65). New cstate: current_term: 1 leader_uuid: "363ff42f68d84e7d95a885b2447673c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:36.435267 3521 consensus_queue.cc:237] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:36.435818 3205 catalog_manager.cc:5484] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67). New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:36.441431 28893 tablet_server.cc:178] TabletServer@127.28.55.65:0 shutting down...
I20240503 08:10:36.444248 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:36.444422 28893 tablet_replica.cc:330] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: stopping tablet replica
I20240503 08:10:36.444497 28893 raft_consensus.cc:2229] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:36.444561 28893 pending_rounds.cc:62] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Trying to abort 1 pending ops.
I20240503 08:10:36.444586 28893 pending_rounds.cc:68] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508834032934912 op_type: NO_OP noop_request { }
I20240503 08:10:36.444649 28893 raft_consensus.cc:2866] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:10:36.444677 28893 raft_consensus.cc:2258] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:36.444957 28893 tablet_replica.cc:330] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: stopping tablet replica
I20240503 08:10:36.445010 28893 raft_consensus.cc:2229] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:36.445050 28893 raft_consensus.cc:2258] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:36.445225 28893 tablet_replica.cc:330] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: stopping tablet replica
I20240503 08:10:36.445281 28893 raft_consensus.cc:2229] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:36.445315 28893 raft_consensus.cc:2258] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:36.445462 28893 tablet_replica.cc:330] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: stopping tablet replica
I20240503 08:10:36.445498 28893 raft_consensus.cc:2229] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:36.445531 28893 raft_consensus.cc:2258] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:36.445662 28893 tablet_replica.cc:330] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: stopping tablet replica
I20240503 08:10:36.445701 28893 raft_consensus.cc:2229] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:36.445734 28893 raft_consensus.cc:2258] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:36.445864 28893 tablet_replica.cc:330] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: stopping tablet replica
I20240503 08:10:36.445900 28893 raft_consensus.cc:2229] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:36.445945 28893 pending_rounds.cc:62] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Trying to abort 1 pending ops.
I20240503 08:10:36.445969 28893 pending_rounds.cc:68] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508833048293376 op_type: NO_OP noop_request { }
I20240503 08:10:36.446017 28893 raft_consensus.cc:2866] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:10:36.446046 28893 raft_consensus.cc:2258] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:36.446276 28893 tablet_replica.cc:330] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: stopping tablet replica
I20240503 08:10:36.446314 28893 raft_consensus.cc:2229] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:36.446345 28893 raft_consensus.cc:2258] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:36.446508 28893 tablet_replica.cc:330] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: stopping tablet replica
I20240503 08:10:36.446543 28893 raft_consensus.cc:2229] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:36.446563 28893 raft_consensus.cc:2258] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:36.459072 28893 tablet_server.cc:194] TabletServer@127.28.55.65:0 shutdown complete.
I20240503 08:10:36.618677 3515 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:36.620605 3425 consensus_peers.cc:485] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): Couldn't send request to peer 363ff42f68d84e7d95a885b2447673c2. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:37655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:10:36.680961 3425 consensus_peers.cc:485] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): Couldn't send request to peer 363ff42f68d84e7d95a885b2447673c2. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:37655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:36.683211 3521 consensus_queue.cc:1035] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:36.811801 3521 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:36.813548 3425 consensus_peers.cc:485] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): Couldn't send request to peer 363ff42f68d84e7d95a885b2447673c2. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:37655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:10:36.893751 3425 consensus_peers.cc:485] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): Couldn't send request to peer 363ff42f68d84e7d95a885b2447673c2. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:37655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:36.896637 3560 consensus_queue.cc:1035] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:36.999760 3350 consensus_peers.cc:485] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 -> Peer 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): Couldn't send request to peer 363ff42f68d84e7d95a885b2447673c2. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:37655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:37.024655 3512 consensus_queue.cc:1035] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:37.046644 3425 consensus_peers.cc:485] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): Couldn't send request to peer 363ff42f68d84e7d95a885b2447673c2. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:37655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:37.052726 3560 consensus_queue.cc:1035] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:37.198323 3521 consensus_queue.cc:579] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Leader has been unable to successfully communicate with peer 363ff42f68d84e7d95a885b2447673c2 for more than 1 seconds (1.031s)
I20240503 08:10:37.288079 3570 consensus_queue.cc:579] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Leader has been unable to successfully communicate with peer 363ff42f68d84e7d95a885b2447673c2 for more than 1 seconds (1.086s)
I20240503 08:10:37.425889 3570 consensus_queue.cc:579] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Leader has been unable to successfully communicate with peer 363ff42f68d84e7d95a885b2447673c2 for more than 1 seconds (1.079s)
I20240503 08:10:37.464054 3204 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48540:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240503 08:10:37.464161 3204 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48540:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240503 08:10:37.465302 3204 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b: Sending DeleteTablet for 3 replicas of tablet fe08e57f721a4ffe96c6311e34195923
I20240503 08:10:37.465415 3204 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b: Sending DeleteTablet for 3 replicas of tablet a5bfef61532441b7b07943a3797f9a5c
I20240503 08:10:37.465468 3204 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b: Sending DeleteTablet for 3 replicas of tablet 20cfe08358ea45a893b7c5dcc5ebeb4c
I20240503 08:10:37.465502 3204 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b: Sending DeleteTablet for 3 replicas of tablet 341bb356ba2d41c2a6f0fd5c36260645
I20240503 08:10:37.465615 3373 tablet_service.cc:1514] Processing DeleteTablet for tablet a5bfef61532441b7b07943a3797f9a5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:59470
I20240503 08:10:37.465636 3371 tablet_service.cc:1514] Processing DeleteTablet for tablet fe08e57f721a4ffe96c6311e34195923 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:59470
I20240503 08:10:37.465713 3372 tablet_service.cc:1514] Processing DeleteTablet for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:59470
I20240503 08:10:37.465834 3448 tablet_service.cc:1514] Processing DeleteTablet for tablet a5bfef61532441b7b07943a3797f9a5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:51372
I20240503 08:10:37.465906 3446 tablet_service.cc:1514] Processing DeleteTablet for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:51372
I20240503 08:10:37.465977 3449 tablet_service.cc:1514] Processing DeleteTablet for tablet fe08e57f721a4ffe96c6311e34195923 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:51372
I20240503 08:10:37.465962 3447 tablet_service.cc:1514] Processing DeleteTablet for tablet 341bb356ba2d41c2a6f0fd5c36260645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:51372
I20240503 08:10:37.466061 3372 tablet_service.cc:1514] Processing DeleteTablet for tablet 341bb356ba2d41c2a6f0fd5c36260645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:59470
I20240503 08:10:37.466104 3580 tablet_replica.cc:330] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: stopping tablet replica
I20240503 08:10:37.465957 3578 tablet_replica.cc:330] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: stopping tablet replica
I20240503 08:10:37.466158 3578 raft_consensus.cc:2229] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:37.466157 3580 raft_consensus.cc:2229] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:37.466228 3578 raft_consensus.cc:2258] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.466269 3580 raft_consensus.cc:2258] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.466307 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:37.466528 3578 ts_tablet_manager.cc:1875] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:10:37.467176 3192 catalog_manager.cc:4573] TS 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5bfef61532441b7b07943a3797f9a5c: Network error: Client connection negotiation failed: client connection to 127.28.55.65:37655: connect: Connection refused (error 111)
I20240503 08:10:37.467254 3580 ts_tablet_manager.cc:1875] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:10:37.467520 3581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:37.467764 28893 server_base.cc:1034] running on GCE node
I20240503 08:10:37.468528 3578 ts_tablet_manager.cc:1888] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 08:10:37.468580 3578 log.cc:1196] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/a5bfef61532441b7b07943a3797f9a5c
I20240503 08:10:37.468744 3580 ts_tablet_manager.cc:1888] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 08:10:37.468770 3578 ts_tablet_manager.cc:1909] T a5bfef61532441b7b07943a3797f9a5c P ad402dfffae4424eb09cef218f971013: Deleting consensus metadata
I20240503 08:10:37.468796 3580 log.cc:1196] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/a5bfef61532441b7b07943a3797f9a5c
I20240503 08:10:37.468987 3580 ts_tablet_manager.cc:1909] T a5bfef61532441b7b07943a3797f9a5c P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting consensus metadata
I20240503 08:10:37.469161 3578 tablet_replica.cc:330] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: stopping tablet replica
I20240503 08:10:37.469210 3580 tablet_replica.cc:330] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: stopping tablet replica
I20240503 08:10:37.469259 3580 raft_consensus.cc:2229] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:37.469246 3189 catalog_manager.cc:4830] TS ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797): tablet a5bfef61532441b7b07943a3797f9a5c (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.469347 3578 raft_consensus.cc:2229] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:37.469309 3580 raft_consensus.cc:2258] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:10:37.469318 3582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:37.469302 3192 catalog_manager.cc:4830] TS 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655): tablet a5bfef61532441b7b07943a3797f9a5c (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
W20240503 08:10:37.469226 3584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:37.469424 3578 raft_consensus.cc:2258] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.469722 3578 ts_tablet_manager.cc:1875] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.469748 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:37.469772 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:37.469787 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723837469787 us; error 0 us; skew 500 ppm
I20240503 08:10:37.470130 3580 ts_tablet_manager.cc:1875] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.470656 28893 webserver.cc:460] Webserver started at http://127.28.55.68:42113/ using document root <none> and password file <none>
I20240503 08:10:37.470741 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:37.470782 28893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:37.470851 28893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:37.471073 3578 ts_tablet_manager.cc:1888] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 08:10:37.471098 28893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-3-root/instance:
uuid: "cb463629df14422090d3167cc3b496ed"
format_stamp: "Formatted at 2024-05-03 08:10:37 on dist-test-slave-mlgg"
I20240503 08:10:37.471133 3578 log.cc:1196] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/fe08e57f721a4ffe96c6311e34195923
I20240503 08:10:37.471307 3578 ts_tablet_manager.cc:1909] T fe08e57f721a4ffe96c6311e34195923 P ad402dfffae4424eb09cef218f971013: Deleting consensus metadata
I20240503 08:10:37.471416 3580 ts_tablet_manager.cc:1888] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 08:10:37.471464 3580 log.cc:1196] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/20cfe08358ea45a893b7c5dcc5ebeb4c
I20240503 08:10:37.471622 3580 ts_tablet_manager.cc:1909] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting consensus metadata
I20240503 08:10:37.471743 3189 catalog_manager.cc:4830] TS ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797): tablet fe08e57f721a4ffe96c6311e34195923 (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.471832 3578 tablet_replica.cc:330] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: stopping tablet replica
I20240503 08:10:37.471876 3578 raft_consensus.cc:2229] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:37.471954 3578 raft_consensus.cc:2258] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.472036 3580 tablet_replica.cc:330] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: stopping tablet replica
I20240503 08:10:37.472116 3192 catalog_manager.cc:4830] TS 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655): tablet 20cfe08358ea45a893b7c5dcc5ebeb4c (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.472106 3580 raft_consensus.cc:2229] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:37.472203 28893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:37.472203 3580 raft_consensus.cc:2258] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.472309 3578 ts_tablet_manager.cc:1875] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.472378 3580 ts_tablet_manager.cc:1875] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.472741 3589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:37.472843 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:37.472879 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-3-root
uuid: "cb463629df14422090d3167cc3b496ed"
format_stamp: "Formatted at 2024-05-03 08:10:37 on dist-test-slave-mlgg"
I20240503 08:10:37.472921 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:37.473512 3578 ts_tablet_manager.cc:1888] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 08:10:37.473569 3578 log.cc:1196] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/20cfe08358ea45a893b7c5dcc5ebeb4c
I20240503 08:10:37.473587 3580 ts_tablet_manager.cc:1888] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 08:10:37.473641 3580 log.cc:1196] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/fe08e57f721a4ffe96c6311e34195923
I20240503 08:10:37.473754 3578 ts_tablet_manager.cc:1909] T 20cfe08358ea45a893b7c5dcc5ebeb4c P ad402dfffae4424eb09cef218f971013: Deleting consensus metadata
I20240503 08:10:37.473790 3580 ts_tablet_manager.cc:1909] T fe08e57f721a4ffe96c6311e34195923 P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting consensus metadata
I20240503 08:10:37.474166 3580 tablet_replica.cc:330] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: stopping tablet replica
I20240503 08:10:37.474215 3580 raft_consensus.cc:2229] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:37.474259 3192 catalog_manager.cc:4830] TS 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655): tablet fe08e57f721a4ffe96c6311e34195923 (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.474282 3578 tablet_replica.cc:330] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: stopping tablet replica
I20240503 08:10:37.474298 3580 raft_consensus.cc:2258] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.474362 3578 raft_consensus.cc:2229] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:37.474259 3189 catalog_manager.cc:4830] TS ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797): tablet 20cfe08358ea45a893b7c5dcc5ebeb4c (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.474506 3580 ts_tablet_manager.cc:1875] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.474527 3578 raft_consensus.cc:2258] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.474831 3578 ts_tablet_manager.cc:1875] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.475952 3580 ts_tablet_manager.cc:1888] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 08:10:37.475960 3578 ts_tablet_manager.cc:1888] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 08:10:37.476024 3580 log.cc:1196] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/341bb356ba2d41c2a6f0fd5c36260645
I20240503 08:10:37.476145 3578 log.cc:1196] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/341bb356ba2d41c2a6f0fd5c36260645
I20240503 08:10:37.476329 3578 ts_tablet_manager.cc:1909] T 341bb356ba2d41c2a6f0fd5c36260645 P ad402dfffae4424eb09cef218f971013: Deleting consensus metadata
I20240503 08:10:37.476332 3580 ts_tablet_manager.cc:1909] T 341bb356ba2d41c2a6f0fd5c36260645 P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting consensus metadata
I20240503 08:10:37.476652 3192 catalog_manager.cc:4830] TS 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655): tablet 341bb356ba2d41c2a6f0fd5c36260645 (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.476651 3189 catalog_manager.cc:4830] TS ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797): tablet 341bb356ba2d41c2a6f0fd5c36260645 (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.483433 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:37.483618 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:37.483745 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:37.483933 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:37.483963 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:37.483986 28893 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:37.483999 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:37.486752 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.68:39211
I20240503 08:10:37.487282 28893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:37.487412 3652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.68:39211 every 8 connection(s)
W20240503 08:10:37.488523 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:37.488687 28893 server_base.cc:1034] running on GCE node
W20240503 08:10:37.488930 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:37.489111 3659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:37.489248 28893 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:37.489293 28893 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:37.489310 28893 hybrid_clock.cc:648] HybridClock initialized: now 1714723837489310 us; error 0 us; skew 500 ppm
I20240503 08:10:37.490101 28893 webserver.cc:460] Webserver started at http://127.28.55.65:41825/ using document root <none> and password file <none>
I20240503 08:10:37.490185 28893 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:37.490221 28893 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:10:37.490944 28893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:37.491073 3653 heartbeater.cc:347] Connected to a master server at 127.28.55.126:35865
I20240503 08:10:37.491130 3653 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:37.491252 3653 heartbeater.cc:511] Master 127.28.55.126:35865 requested a full tablet report, sending...
I20240503 08:10:37.491488 3204 ts_manager.cc:194] Registered new tserver with Master: cb463629df14422090d3167cc3b496ed (127.28.55.68:39211)
I20240503 08:10:37.491788 3664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:37.491906 28893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:37.491936 3204 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48558
I20240503 08:10:37.491948 28893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root
uuid: "363ff42f68d84e7d95a885b2447673c2"
format_stamp: "Formatted at 2024-05-03 08:10:35 on dist-test-slave-mlgg"
I20240503 08:10:37.492074 28893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:37.512933 28893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:37.513168 28893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:37.513299 28893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:37.513590 3671 ts_tablet_manager.cc:516] Loading tablet metadata (0/8 complete)
I20240503 08:10:37.516037 28893 ts_tablet_manager.cc:559] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20240503 08:10:37.516106 28893 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 08:10:37.516134 28893 ts_tablet_manager.cc:574] Registering tablets (0/8 complete)
I20240503 08:10:37.516543 3671 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:37.517630 3671 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:37.517813 3671 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap complete.
I20240503 08:10:37.517933 3671 ts_tablet_manager.cc:1367] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:37.518070 3671 raft_consensus.cc:348] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:37.518154 3671 raft_consensus.cc:729] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:37.518205 3671 consensus_queue.cc:260] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:37.518260 28893 ts_tablet_manager.cc:590] Registered 8 tablets
I20240503 08:10:37.518297 28893 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240503 08:10:37.518388 3671 ts_tablet_manager.cc:1398] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:37.518455 3671 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:37.519424 3671 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:37.519600 3671 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap complete.
I20240503 08:10:37.519714 3671 ts_tablet_manager.cc:1367] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:37.519869 3671 raft_consensus.cc:348] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.519946 3671 raft_consensus.cc:729] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:37.520009 3671 consensus_queue.cc:260] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.520112 3671 ts_tablet_manager.cc:1398] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:37.520169 3671 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:37.521219 3459 consensus_queue.cc:237] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:37.521342 3458 consensus_queue.cc:237] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:37.521934 3384 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:37.522010 3383 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:37.522046 3671 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:10:37.522176 3570 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:10:37.522254 3671 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Bootstrap complete.
I20240503 08:10:37.522301 3570 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:10:37.522325 3732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.55.65:37655 every 8 connection(s)
I20240503 08:10:37.522344 3671 ts_tablet_manager.cc:1367] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:10:37.521939 28893 rpc_server.cc:307] RPC server started. Bound to: 127.28.55.65:37655
I20240503 08:10:37.522619 3671 raft_consensus.cc:348] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.522701 3671 raft_consensus.cc:729] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:37.522745 3671 consensus_queue.cc:260] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.522902 3671 ts_tablet_manager.cc:1398] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:37.522955 3671 tablet_bootstrap.cc:492] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:37.522954 3515 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.524092 3192 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 85bb842493534b7cbc575a3fcf47a4c1 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20240503 08:10:37.524345 3570 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.524410 3204 catalog_manager.cc:5484] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e reported cstate change: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:37.524650 3384 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.526261 3192 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1a864e641f254544a457ddc37f4bba26 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
I20240503 08:10:37.526410 3384 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.526376 3706 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:37.526381 3704 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:37.526697 3560 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:37.527221 3671 tablet_bootstrap.cc:492] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:37.527395 3671 tablet_bootstrap.cc:492] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap complete.
I20240503 08:10:37.527508 3671 ts_tablet_manager.cc:1367] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:10:37.527403 3704 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.527662 3671 raft_consensus.cc:348] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.527658 3204 catalog_manager.cc:5484] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e reported cstate change: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:37.527834 3671 raft_consensus.cc:729] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:37.527875 3671 consensus_queue.cc:260] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.527505 3515 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:37.527952 3671 ts_tablet_manager.cc:1398] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:37.528467 3671 tablet_bootstrap.cc:492] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:37.528981 3706 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.529202 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:48784
W20240503 08:10:37.529522 3423 consensus_peers.cc:485] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer cb463629df14422090d3167cc3b496ed (127.28.55.68:39211): Couldn't send request to peer cb463629df14422090d3167cc3b496ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a864e641f254544a457ddc37f4bba26. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:10:37.529595 3423 consensus_peers.cc:485] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer cb463629df14422090d3167cc3b496ed (127.28.55.68:39211): Couldn't send request to peer cb463629df14422090d3167cc3b496ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85bb842493534b7cbc575a3fcf47a4c1. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:10:37.529592 3192 catalog_manager.cc:4810] TS 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): delete failed for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20cfe08358ea45a893b7c5dcc5ebeb4c already in progress: opening tablet
I20240503 08:10:37.530014 3671 tablet_bootstrap.cc:492] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:37.530182 3671 tablet_bootstrap.cc:492] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Bootstrap complete.
I20240503 08:10:37.530285 3671 ts_tablet_manager.cc:1367] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:37.530452 3671 raft_consensus.cc:348] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.530511 3671 raft_consensus.cc:729] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:37.530561 3671 consensus_queue.cc:260] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.530649 3671 ts_tablet_manager.cc:1398] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:37.530704 3671 tablet_bootstrap.cc:492] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:37.531919 3671 tablet_bootstrap.cc:492] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:10:37.532125 3671 tablet_bootstrap.cc:492] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap complete.
I20240503 08:10:37.532224 3671 ts_tablet_manager.cc:1367] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:37.532374 3671 raft_consensus.cc:348] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.532444 3671 raft_consensus.cc:729] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:37.532456 3736 heartbeater.cc:347] Connected to a master server at 127.28.55.126:35865
I20240503 08:10:37.532495 3736 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:37.532485 3671 consensus_queue.cc:260] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.532572 3671 ts_tablet_manager.cc:1398] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:37.532603 3736 heartbeater.cc:511] Master 127.28.55.126:35865 requested a full tablet report, sending...
I20240503 08:10:37.532622 3671 tablet_bootstrap.cc:492] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
I20240503 08:10:37.532902 3205 ts_manager.cc:194] Re-registered known tserver with Master: 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655)
I20240503 08:10:37.533238 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:48784
I20240503 08:10:37.533515 3205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48564
W20240503 08:10:37.533588 3192 catalog_manager.cc:4810] TS 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): delete failed for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20cfe08358ea45a893b7c5dcc5ebeb4c already in progress: opening tablet
I20240503 08:10:37.533811 3671 tablet_bootstrap.cc:492] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:37.533974 3736 heartbeater.cc:503] Master 127.28.55.126:35865 was elected leader, sending a full tablet report...
I20240503 08:10:37.533967 3671 tablet_bootstrap.cc:492] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: Bootstrap complete.
I20240503 08:10:37.534116 3671 ts_tablet_manager.cc:1367] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:37.534255 3671 raft_consensus.cc:348] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.534327 3671 raft_consensus.cc:729] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:37.534390 3671 consensus_queue.cc:260] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.534483 3671 ts_tablet_manager.cc:1398] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:37.534507 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:48784
I20240503 08:10:37.534539 3671 tablet_bootstrap.cc:492] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Bootstrap starting.
W20240503 08:10:37.534950 3192 catalog_manager.cc:4810] TS 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): delete failed for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20cfe08358ea45a893b7c5dcc5ebeb4c already in progress: opening tablet
I20240503 08:10:37.535647 3671 tablet_bootstrap.cc:492] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:37.535770 3671 tablet_bootstrap.cc:492] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Bootstrap complete.
I20240503 08:10:37.535848 3671 ts_tablet_manager.cc:1367] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:37.535960 3671 raft_consensus.cc:348] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.536003 3671 raft_consensus.cc:729] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ff42f68d84e7d95a885b2447673c2, State: Initialized, Role: FOLLOWER
I20240503 08:10:37.536039 3671 consensus_queue.cc:260] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.536144 3671 ts_tablet_manager.cc:1398] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:37.536540 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet 20cfe08358ea45a893b7c5dcc5ebeb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:48784
I20240503 08:10:37.536640 3747 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:37.536690 3747 raft_consensus.cc:2229] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:37.536724 3747 raft_consensus.cc:2258] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.536916 3747 ts_tablet_manager.cc:1875] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.538232 3747 ts_tablet_manager.cc:1888] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 08:10:37.538280 3747 log.cc:1196] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/20cfe08358ea45a893b7c5dcc5ebeb4c
I20240503 08:10:37.538416 3747 ts_tablet_manager.cc:1909] T 20cfe08358ea45a893b7c5dcc5ebeb4c P 363ff42f68d84e7d95a885b2447673c2: Deleting consensus metadata
I20240503 08:10:37.538722 3192 catalog_manager.cc:4830] TS 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): tablet 20cfe08358ea45a893b7c5dcc5ebeb4c (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.541121 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet 341bb356ba2d41c2a6f0fd5c36260645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:48784
I20240503 08:10:37.541234 3747 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:37.541290 3747 raft_consensus.cc:2229] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:37.541337 3747 raft_consensus.cc:2258] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.541527 3747 ts_tablet_manager.cc:1875] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.542645 3747 ts_tablet_manager.cc:1888] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 08:10:37.542694 3747 log.cc:1196] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/341bb356ba2d41c2a6f0fd5c36260645
I20240503 08:10:37.542874 3747 ts_tablet_manager.cc:1909] T 341bb356ba2d41c2a6f0fd5c36260645 P 363ff42f68d84e7d95a885b2447673c2: Deleting consensus metadata
I20240503 08:10:37.543228 3192 catalog_manager.cc:4830] TS 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): tablet 341bb356ba2d41c2a6f0fd5c36260645 (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.543388 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet a5bfef61532441b7b07943a3797f9a5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:48784
I20240503 08:10:37.543483 3747 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:37.543543 3747 raft_consensus.cc:2229] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:37.543581 3747 raft_consensus.cc:2258] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.543741 3747 ts_tablet_manager.cc:1875] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.544852 3747 ts_tablet_manager.cc:1888] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240503 08:10:37.544898 3747 log.cc:1196] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/a5bfef61532441b7b07943a3797f9a5c
I20240503 08:10:37.545032 3747 ts_tablet_manager.cc:1909] T a5bfef61532441b7b07943a3797f9a5c P 363ff42f68d84e7d95a885b2447673c2: Deleting consensus metadata
I20240503 08:10:37.545302 3192 catalog_manager.cc:4830] TS 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): tablet a5bfef61532441b7b07943a3797f9a5c (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.547695 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet fe08e57f721a4ffe96c6311e34195923 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:10:37 UTC) from {username='slave'} at 127.0.0.1:48784
I20240503 08:10:37.547806 3747 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:37.547873 3747 raft_consensus.cc:2229] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:37.547914 3747 pending_rounds.cc:62] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Trying to abort 1 pending ops.
I20240503 08:10:37.547940 3747 pending_rounds.cc:68] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023508834032934912 op_type: NO_OP noop_request { }
I20240503 08:10:37.547993 3747 raft_consensus.cc:2866] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:10:37.548020 3747 raft_consensus.cc:2258] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:37.548236 3747 ts_tablet_manager.cc:1875] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:10:37.549422 3747 ts_tablet_manager.cc:1888] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240503 08:10:37.549470 3747 log.cc:1196] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/fe08e57f721a4ffe96c6311e34195923
I20240503 08:10:37.549633 3747 ts_tablet_manager.cc:1909] T fe08e57f721a4ffe96c6311e34195923 P 363ff42f68d84e7d95a885b2447673c2: Deleting consensus metadata
I20240503 08:10:37.549927 3192 catalog_manager.cc:4830] TS 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): tablet fe08e57f721a4ffe96c6311e34195923 (table dugtrio [id=1765e08367a641ff919f78e7506340a5]) successfully deleted
I20240503 08:10:37.597112 3458 consensus_queue.cc:237] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:37.597724 3384 raft_consensus.cc:1261] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:37.597748 3706 raft_consensus.cc:1261] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:10:37.597837 3423 consensus_peers.cc:485] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer cb463629df14422090d3167cc3b496ed (127.28.55.68:39211): Couldn't send request to peer cb463629df14422090d3167cc3b496ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bb52224259f4909a1838a56ba8121a9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:37.597986 3560 consensus_queue.cc:1035] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:10:37.598100 3521 consensus_queue.cc:1035] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:37.598634 3521 raft_consensus.cc:2932] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.598726 3384 raft_consensus.cc:2932] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.599679 3192 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6bb52224259f4909a1838a56ba8121a9 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20240503 08:10:37.599963 3204 catalog_manager.cc:5484] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 reported cstate change: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.599975 3706 raft_consensus.cc:2932] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:37.601917 3458 consensus_queue.cc:237] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.602299 3706 raft_consensus.cc:1261] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:37.602299 3384 raft_consensus.cc:1261] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:37.602507 3560 consensus_queue.cc:1035] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:37.602582 3521 consensus_queue.cc:1035] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:37.603248 3521 raft_consensus.cc:2932] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } }
I20240503 08:10:37.603377 3384 raft_consensus.cc:2932] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } }
I20240503 08:10:37.604153 3192 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6bb52224259f4909a1838a56ba8121a9 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:37.604282 3706 raft_consensus.cc:2932] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } }
I20240503 08:10:37.604350 3205 catalog_manager.cc:5484] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 reported cstate change: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } }
I20240503 08:10:37.607589 3617 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb52224259f4909a1838a56ba8121a9 with delete_type TABLET_DATA_TOMBSTONED (TS cb463629df14422090d3167cc3b496ed not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38300
W20240503 08:10:37.607923 3190 catalog_manager.cc:4797] TS cb463629df14422090d3167cc3b496ed (127.28.55.68:39211): delete failed for tablet 6bb52224259f4909a1838a56ba8121a9 because the tablet was not found. No further retry: Not found: Tablet not found: 6bb52224259f4909a1838a56ba8121a9
I20240503 08:10:37.953291 3458 consensus_queue.cc:237] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.953802 3384 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:37.953802 3706 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:37.954056 3570 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:37.954182 3515 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:37.954893 3570 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } }
I20240503 08:10:37.955216 3512 raft_consensus.cc:482] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:37.955271 3512 raft_consensus.cc:504] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.955302 3384 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } }
I20240503 08:10:37.955382 3512 leader_election.cc:288] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:37.955868 3706 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } }
I20240503 08:10:37.956290 3458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b615b62f5f84fa387d8d07f6f2a1ade" candidate_uuid: "ad402dfffae4424eb09cef218f971013" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" is_pre_election: true
I20240503 08:10:37.956393 3458 raft_consensus.cc:2454] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad402dfffae4424eb09cef218f971013 in term 1.
I20240503 08:10:37.956681 3350 leader_election.cc:302] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbefbfd77e7474a8a69b1c536fde51e, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:37.956898 3192 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a864e641f254544a457ddc37f4bba26 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:37.957046 3512 raft_consensus.cc:2781] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:10:37.957087 3512 raft_consensus.cc:482] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:37.957104 3512 raft_consensus.cc:3037] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:37.957316 3203 catalog_manager.cc:5484] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 reported cstate change: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } }
I20240503 08:10:37.957758 3512 raft_consensus.cc:504] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.957873 3512 leader_election.cc:288] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [CANDIDATE]: Term 2 election: Requested vote from peers 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655), 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:37.958186 3458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b615b62f5f84fa387d8d07f6f2a1ade" candidate_uuid: "ad402dfffae4424eb09cef218f971013" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbefbfd77e7474a8a69b1c536fde51e"
I20240503 08:10:37.958252 3458 raft_consensus.cc:3037] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:37.958941 3458 raft_consensus.cc:2454] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad402dfffae4424eb09cef218f971013 in term 2.
I20240503 08:10:37.959086 3350 leader_election.cc:302] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbefbfd77e7474a8a69b1c536fde51e, ad402dfffae4424eb09cef218f971013; no voters:
I20240503 08:10:37.959215 3754 raft_consensus.cc:2781] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:37.959280 3754 raft_consensus.cc:686] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 2 LEADER]: Becoming Leader. State: Replica: ad402dfffae4424eb09cef218f971013, State: Running, Role: LEADER
I20240503 08:10:37.959328 3754 consensus_queue.cc:237] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:37.959566 3706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b615b62f5f84fa387d8d07f6f2a1ade" candidate_uuid: "ad402dfffae4424eb09cef218f971013" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2" is_pre_election: true
I20240503 08:10:37.959664 3706 raft_consensus.cc:2396] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ad402dfffae4424eb09cef218f971013 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 08:10:37.959579 3704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b615b62f5f84fa387d8d07f6f2a1ade" candidate_uuid: "ad402dfffae4424eb09cef218f971013" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ff42f68d84e7d95a885b2447673c2"
I20240503 08:10:37.959751 3704 raft_consensus.cc:3037] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:37.959916 3205 catalog_manager.cc:5484] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 reported cstate change: term changed from 1 to 2, leader changed from 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65) to ad402dfffae4424eb09cef218f971013 (127.28.55.66). New cstate: current_term: 2 leader_uuid: "ad402dfffae4424eb09cef218f971013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:37.960444 3704 raft_consensus.cc:2396] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ad402dfffae4424eb09cef218f971013 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 08:10:37.960534 3617 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a864e641f254544a457ddc37f4bba26 with delete_type TABLET_DATA_TOMBSTONED (TS cb463629df14422090d3167cc3b496ed not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38300
W20240503 08:10:37.960696 3190 catalog_manager.cc:4797] TS cb463629df14422090d3167cc3b496ed (127.28.55.68:39211): delete failed for tablet 1a864e641f254544a457ddc37f4bba26 because the tablet was not found. No further retry: Not found: Tablet not found: 1a864e641f254544a457ddc37f4bba26
W20240503 08:10:38.019183 3253 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240503 08:10:38.019717 3253 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8b615b62f5f84fa387d8d07f6f2a1ade, TS 4dbefbfd77e7474a8a69b1c536fde51e -> TS cb463629df14422090d3167cc3b496ed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
W20240503 08:10:38.019762 3253 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 8b615b62f5f84fa387d8d07f6f2a1ade, TS 4dbefbfd77e7474a8a69b1c536fde51e -> TS cb463629df14422090d3167cc3b496ed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
W20240503 08:10:38.020226 3253 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1a864e641f254544a457ddc37f4bba26, TS ad402dfffae4424eb09cef218f971013 -> TS cb463629df14422090d3167cc3b496ed move failed, destination replica disappeared from tablet's Raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
W20240503 08:10:38.020264 3253 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 1a864e641f254544a457ddc37f4bba26, TS ad402dfffae4424eb09cef218f971013 -> TS cb463629df14422090d3167cc3b496ed move failed, destination replica disappeared from tablet's Raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
W20240503 08:10:38.020557 3253 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6bb52224259f4909a1838a56ba8121a9, TS 363ff42f68d84e7d95a885b2447673c2 -> TS cb463629df14422090d3167cc3b496ed move failed, destination replica disappeared from tablet's Raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
W20240503 08:10:38.020596 3253 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 6bb52224259f4909a1838a56ba8121a9, TS 363ff42f68d84e7d95a885b2447673c2 -> TS cb463629df14422090d3167cc3b496ed move failed, destination replica disappeared from tablet's Raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } }
I20240503 08:10:38.060065 3758 ts_tablet_manager.cc:902] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: Initiating tablet copy from peer 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:38.060377 3758 tablet_copy_client.cc:308] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.67:38655
I20240503 08:10:38.061635 3458 consensus_queue.cc:237] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:38.062073 3384 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:38.062076 3704 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:10:38.062276 3560 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:38.062366 3521 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:38.063030 3570 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } }
I20240503 08:10:38.063174 3384 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } }
I20240503 08:10:38.064162 3192 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 85bb842493534b7cbc575a3fcf47a4c1 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:38.064147 3704 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } }
I20240503 08:10:38.064148 3469 tablet_copy_service.cc:140] P 4dbefbfd77e7474a8a69b1c536fde51e: Received BeginTabletCopySession request for tablet 85bb842493534b7cbc575a3fcf47a4c1 from peer cb463629df14422090d3167cc3b496ed ({username='slave'} at 127.0.0.1:48940)
I20240503 08:10:38.064282 3469 tablet_copy_service.cc:161] P 4dbefbfd77e7474a8a69b1c536fde51e: Beginning new tablet copy session on tablet 85bb842493534b7cbc575a3fcf47a4c1 from peer cb463629df14422090d3167cc3b496ed at {username='slave'} at 127.0.0.1:48940: session id = cb463629df14422090d3167cc3b496ed-85bb842493534b7cbc575a3fcf47a4c1
I20240503 08:10:38.064750 3469 tablet_copy_source_session.cc:205] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:38.065011 3205 catalog_manager.cc:5484] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 reported cstate change: config changed from index 2 to 3, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) evicted. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } }
I20240503 08:10:38.065068 3758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85bb842493534b7cbc575a3fcf47a4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:38.065814 3617 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bb842493534b7cbc575a3fcf47a4c1 with delete_type TABLET_DATA_TOMBSTONED (TS cb463629df14422090d3167cc3b496ed not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38300
W20240503 08:10:38.066030 3190 catalog_manager.cc:4797] TS cb463629df14422090d3167cc3b496ed (127.28.55.68:39211): delete failed for tablet 85bb842493534b7cbc575a3fcf47a4c1 because the tablet was not found. No further retry: Not found: Tablet not found: 85bb842493534b7cbc575a3fcf47a4c1
I20240503 08:10:38.066332 3758 tablet_copy_client.cc:783] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:38.066442 3758 tablet_copy_client.cc:647] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:38.066951 3758 tablet_copy_client.cc:523] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:38.067461 3758 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: Bootstrap starting.
I20240503 08:10:38.068673 3758 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:38.068822 3758 tablet_bootstrap.cc:492] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: Bootstrap complete.
I20240503 08:10:38.068981 3758 ts_tablet_manager.cc:1367] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:38.069140 3758 raft_consensus.cc:348] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:38.069195 3758 raft_consensus.cc:729] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: cb463629df14422090d3167cc3b496ed, State: Initialized, Role: NON_PARTICIPANT
I20240503 08:10:38.069223 3758 consensus_queue.cc:260] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } }
I20240503 08:10:38.069406 3758 ts_tablet_manager.cc:1398] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:38.069463 3653 heartbeater.cc:503] Master 127.28.55.126:35865 was elected leader, sending a full tablet report...
I20240503 08:10:38.069624 3469 tablet_copy_service.cc:342] P 4dbefbfd77e7474a8a69b1c536fde51e: Request end of tablet copy session cb463629df14422090d3167cc3b496ed-85bb842493534b7cbc575a3fcf47a4c1 received from {username='slave'} at 127.0.0.1:48940
I20240503 08:10:38.069670 3469 tablet_copy_service.cc:433] P 4dbefbfd77e7474a8a69b1c536fde51e: ending tablet copy session cb463629df14422090d3167cc3b496ed-85bb842493534b7cbc575a3fcf47a4c1 on tablet 85bb842493534b7cbc575a3fcf47a4c1 with peer cb463629df14422090d3167cc3b496ed
I20240503 08:10:38.354651 3704 raft_consensus.cc:1261] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Refusing update from remote peer ad402dfffae4424eb09cef218f971013: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240503 08:10:38.354749 3704 pending_rounds.cc:76] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Aborting all ops after (but not including) 0
I20240503 08:10:38.354763 3704 pending_rounds.cc:97] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240503 08:10:38.354777 3704 raft_consensus.cc:2866] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:10:38.355027 3754 consensus_queue.cc:1035] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:38.356179 3458 raft_consensus.cc:1261] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 2 FOLLOWER]: Refusing update from remote peer ad402dfffae4424eb09cef218f971013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 08:10:38.356444 3769 consensus_queue.cc:1035] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:39.021014 3253 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table 2da1024cb70c41f6a50aa8fd243e5297: could not find any suitable replica to move from server 4dbefbfd77e7474a8a69b1c536fde51e to server cb463629df14422090d3167cc3b496ed
I20240503 08:10:40.024842 3458 consensus_queue.cc:237] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:40.025393 3384 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:10:40.025449 3423 consensus_peers.cc:485] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer cb463629df14422090d3167cc3b496ed (127.28.55.68:39211): Couldn't send request to peer cb463629df14422090d3167cc3b496ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a864e641f254544a457ddc37f4bba26. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:40.025451 3704 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:10:40.025573 3521 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:40.025825 3521 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:40.026472 3778 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.026590 3384 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.026669 3704 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.027794 3203 catalog_manager.cc:5484] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 reported cstate change: config changed from index 3 to 4, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.027936 3384 consensus_queue.cc:237] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:40.028409 3704 raft_consensus.cc:1261] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Refusing update from remote peer ad402dfffae4424eb09cef218f971013: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:10:40.028671 3458 raft_consensus.cc:1261] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 2 FOLLOWER]: Refusing update from remote peer ad402dfffae4424eb09cef218f971013: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:10:40.028676 3786 consensus_queue.cc:1035] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:10:40.028918 3774 consensus_queue.cc:1035] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:10:40.029421 3774 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.029538 3704 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.030671 3205 catalog_manager.cc:5484] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 reported cstate change: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New cstate: current_term: 2 leader_uuid: "ad402dfffae4424eb09cef218f971013" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:40.030721 3458 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.030871 3459 consensus_queue.cc:237] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:40.031352 3626 raft_consensus.cc:1467] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240503 08:10:40.031405 3626 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 NON_PARTICIPANT]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:10:40.031524 3704 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:10:40.031682 3384 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:10:40.031713 3521 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:40.031934 3779 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:40.032024 3521 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:40.032092 3626 raft_consensus.cc:1467] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240503 08:10:40.032665 3521 raft_consensus.cc:1013] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: attempt to promote peer cb463629df14422090d3167cc3b496ed: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240503 08:10:40.032927 3348 consensus_peers.cc:485] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 -> Peer cb463629df14422090d3167cc3b496ed (127.28.55.68:39211): Couldn't send request to peer cb463629df14422090d3167cc3b496ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b615b62f5f84fa387d8d07f6f2a1ade. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:40.033016 3779 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.033139 3704 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.033192 3384 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.034054 3626 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 LEARNER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.034194 3205 catalog_manager.cc:5484] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 reported cstate change: config changed from index 3 to 4, NON_VOTER cb463629df14422090d3167cc3b496ed (127.28.55.68) added. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } } }
I20240503 08:10:40.035003 3778 raft_consensus.cc:1050] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e: attempting to promote NON_VOTER cb463629df14422090d3167cc3b496ed to VOTER
I20240503 08:10:40.035135 3778 consensus_queue.cc:237] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } }
I20240503 08:10:40.035588 3384 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:10:40.035593 3626 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 LEARNER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:10:40.035749 3704 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:10:40.035799 3799 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:10:40.035905 3799 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:10:40.036007 3779 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:10:40.036664 3778 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:40.036842 3626 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:40.036904 3704 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:40.037793 3384 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:40.037806 3204 catalog_manager.cc:5484] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 reported cstate change: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:40.039530 3458 consensus_queue.cc:237] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } }
I20240503 08:10:40.039849 3704 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:10:40.039920 3626 raft_consensus.cc:1261] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:10:40.040010 3779 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:10:40.040081 3778 consensus_queue.cc:1035] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:10:40.040665 3778 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad402dfffae4424eb09cef218f971013 (127.28.55.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:40.040758 3626 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad402dfffae4424eb09cef218f971013 (127.28.55.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:40.040818 3704 raft_consensus.cc:2932] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad402dfffae4424eb09cef218f971013 (127.28.55.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:40.041715 3205 catalog_manager.cc:5484] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e reported cstate change: config changed from index 5 to 6, VOTER ad402dfffae4424eb09cef218f971013 (127.28.55.66) evicted. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:10:40.041806 3192 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 85bb842493534b7cbc575a3fcf47a4c1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:40.042405 3372 tablet_service.cc:1514] Processing DeleteTablet for tablet 85bb842493534b7cbc575a3fcf47a4c1 with delete_type TABLET_DATA_TOMBSTONED (TS ad402dfffae4424eb09cef218f971013 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59470
I20240503 08:10:40.042645 3800 tablet_replica.cc:330] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013: stopping tablet replica
I20240503 08:10:40.042713 3800 raft_consensus.cc:2229] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:40.042766 3800 raft_consensus.cc:2258] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:40.043033 3800 ts_tablet_manager.cc:1875] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:10:40.044471 3800 ts_tablet_manager.cc:1888] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:10:40.044526 3800 log.cc:1196] T 85bb842493534b7cbc575a3fcf47a4c1 P ad402dfffae4424eb09cef218f971013: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-1-root/wals/85bb842493534b7cbc575a3fcf47a4c1
I20240503 08:10:40.044812 3189 catalog_manager.cc:4830] TS ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797): tablet 85bb842493534b7cbc575a3fcf47a4c1 (table test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]) successfully deleted
I20240503 08:10:40.496060 3801 ts_tablet_manager.cc:902] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: Initiating tablet copy from peer 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655)
I20240503 08:10:40.496470 3801 tablet_copy_client.cc:308] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.67:38655
I20240503 08:10:40.496840 3469 tablet_copy_service.cc:140] P 4dbefbfd77e7474a8a69b1c536fde51e: Received BeginTabletCopySession request for tablet 1a864e641f254544a457ddc37f4bba26 from peer cb463629df14422090d3167cc3b496ed ({username='slave'} at 127.0.0.1:48940)
I20240503 08:10:40.496908 3469 tablet_copy_service.cc:161] P 4dbefbfd77e7474a8a69b1c536fde51e: Beginning new tablet copy session on tablet 1a864e641f254544a457ddc37f4bba26 from peer cb463629df14422090d3167cc3b496ed at {username='slave'} at 127.0.0.1:48940: session id = cb463629df14422090d3167cc3b496ed-1a864e641f254544a457ddc37f4bba26
I20240503 08:10:40.497553 3469 tablet_copy_source_session.cc:205] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:40.497936 3801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a864e641f254544a457ddc37f4bba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:40.499238 3801 tablet_copy_client.cc:783] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:40.499397 3801 tablet_copy_client.cc:647] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:40.499958 3801 tablet_copy_client.cc:523] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:40.500854 3801 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: Bootstrap starting.
I20240503 08:10:40.502406 3801 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:40.502617 3801 tablet_bootstrap.cc:492] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: Bootstrap complete.
I20240503 08:10:40.502733 3801 ts_tablet_manager.cc:1367] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:10:40.502908 3801 raft_consensus.cc:348] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:40.502996 3801 raft_consensus.cc:729] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cb463629df14422090d3167cc3b496ed, State: Initialized, Role: LEARNER
I20240503 08:10:40.503060 3801 consensus_queue.cc:260] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:40.503170 3801 ts_tablet_manager.cc:1398] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:40.503345 3469 tablet_copy_service.cc:342] P 4dbefbfd77e7474a8a69b1c536fde51e: Request end of tablet copy session cb463629df14422090d3167cc3b496ed-1a864e641f254544a457ddc37f4bba26 received from {username='slave'} at 127.0.0.1:48940
I20240503 08:10:40.503399 3469 tablet_copy_service.cc:433] P 4dbefbfd77e7474a8a69b1c536fde51e: ending tablet copy session cb463629df14422090d3167cc3b496ed-1a864e641f254544a457ddc37f4bba26 on tablet 1a864e641f254544a457ddc37f4bba26 with peer cb463629df14422090d3167cc3b496ed
I20240503 08:10:40.569746 3801 ts_tablet_manager.cc:902] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: Initiating tablet copy from peer ad402dfffae4424eb09cef218f971013 (127.28.55.66:44797)
I20240503 08:10:40.570089 3801 tablet_copy_client.cc:308] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: tablet copy: Beginning tablet copy session from remote peer at address 127.28.55.66:44797
I20240503 08:10:40.573637 3394 tablet_copy_service.cc:140] P ad402dfffae4424eb09cef218f971013: Received BeginTabletCopySession request for tablet 8b615b62f5f84fa387d8d07f6f2a1ade from peer cb463629df14422090d3167cc3b496ed ({username='slave'} at 127.0.0.1:37578)
I20240503 08:10:40.573724 3394 tablet_copy_service.cc:161] P ad402dfffae4424eb09cef218f971013: Beginning new tablet copy session on tablet 8b615b62f5f84fa387d8d07f6f2a1ade from peer cb463629df14422090d3167cc3b496ed at {username='slave'} at 127.0.0.1:37578: session id = cb463629df14422090d3167cc3b496ed-8b615b62f5f84fa387d8d07f6f2a1ade
I20240503 08:10:40.574347 3394 tablet_copy_source_session.cc:205] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:10:40.574681 3801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b615b62f5f84fa387d8d07f6f2a1ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:40.576244 3801 tablet_copy_client.cc:783] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: tablet copy: Starting download of 0 data blocks...
I20240503 08:10:40.576368 3801 tablet_copy_client.cc:647] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: tablet copy: Starting download of 1 WAL segments...
I20240503 08:10:40.576912 3801 tablet_copy_client.cc:523] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:10:40.577467 3801 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: Bootstrap starting.
I20240503 08:10:40.578698 3801 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:10:40.578878 3801 tablet_bootstrap.cc:492] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: Bootstrap complete.
I20240503 08:10:40.578972 3801 ts_tablet_manager.cc:1367] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:10:40.579128 3801 raft_consensus.cc:348] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:40.579196 3801 raft_consensus.cc:729] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cb463629df14422090d3167cc3b496ed, State: Initialized, Role: LEARNER
I20240503 08:10:40.579226 3801 consensus_queue.cc:260] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: NON_VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: true } }
I20240503 08:10:40.579321 3801 ts_tablet_manager.cc:1398] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:40.579506 3394 tablet_copy_service.cc:342] P ad402dfffae4424eb09cef218f971013: Request end of tablet copy session cb463629df14422090d3167cc3b496ed-8b615b62f5f84fa387d8d07f6f2a1ade received from {username='slave'} at 127.0.0.1:37578
I20240503 08:10:40.579551 3394 tablet_copy_service.cc:433] P ad402dfffae4424eb09cef218f971013: ending tablet copy session cb463629df14422090d3167cc3b496ed-8b615b62f5f84fa387d8d07f6f2a1ade on tablet 8b615b62f5f84fa387d8d07f6f2a1ade with peer cb463629df14422090d3167cc3b496ed
I20240503 08:10:40.939203 3626 raft_consensus.cc:1203] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:10:41.064356 3626 raft_consensus.cc:1203] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 08:10:41.494210 3521 raft_consensus.cc:1050] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e: attempting to promote NON_VOTER cb463629df14422090d3167cc3b496ed to VOTER
I20240503 08:10:41.494474 3521 consensus_queue.cc:237] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } }
I20240503 08:10:41.495016 3384 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:10:41.495030 3704 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:10:41.495126 3626 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [term 1 LEARNER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:10:41.495235 3818 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:10:41.495414 3819 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:10:41.495548 3521 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:10:41.496251 3521 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.497438 3626 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.497467 3704 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.497495 3384 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.498606 3202 catalog_manager.cc:5484] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed reported cstate change: config changed from index 4 to 5, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } attrs { replace: true } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.500399 3459 consensus_queue.cc:237] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } }
I20240503 08:10:41.500761 3384 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:10:41.500762 3626 raft_consensus.cc:1261] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Refusing update from remote peer 4dbefbfd77e7474a8a69b1c536fde51e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:10:41.500941 3521 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:10:41.501112 3521 consensus_queue.cc:1035] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:10:41.501466 3818 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.501585 3384 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.502573 3205 catalog_manager.cc:5484] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 reported cstate change: config changed from index 5 to 6, VOTER 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65) evicted. New cstate: current_term: 1 leader_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.502632 3192 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a864e641f254544a457ddc37f4bba26 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:41.502708 3815 raft_consensus.cc:1050] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013: attempting to promote NON_VOTER cb463629df14422090d3167cc3b496ed to VOTER
I20240503 08:10:41.502884 3815 consensus_queue.cc:237] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } }
I20240503 08:10:41.503352 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a864e641f254544a457ddc37f4bba26 with delete_type TABLET_DATA_TOMBSTONED (TS 363ff42f68d84e7d95a885b2447673c2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48784
I20240503 08:10:41.503537 3704 raft_consensus.cc:1261] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Refusing update from remote peer ad402dfffae4424eb09cef218f971013: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:41.503685 3459 raft_consensus.cc:1261] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 2 FOLLOWER]: Refusing update from remote peer ad402dfffae4424eb09cef218f971013: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:41.503716 3829 consensus_queue.cc:1035] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:41.503856 3793 consensus_queue.cc:1035] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:41.504082 3626 raft_consensus.cc:2932] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.505115 3831 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:41.505122 3626 raft_consensus.cc:1261] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [term 2 LEARNER]: Refusing update from remote peer ad402dfffae4424eb09cef218f971013: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:10:41.505198 3831 raft_consensus.cc:2229] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:41.505400 3830 consensus_queue.cc:1035] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:10:41.505448 3831 raft_consensus.cc:2258] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:41.505661 3459 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.505534 3830 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.506109 3831 ts_tablet_manager.cc:1875] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:10:41.506363 3626 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.506913 3203 catalog_manager.cc:5484] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 reported cstate change: config changed from index 2 to 3, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ad402dfffae4424eb09cef218f971013" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:10:41.507469 3704 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, cb463629df14422090d3167cc3b496ed (127.28.55.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "4dbefbfd77e7474a8a69b1c536fde51e" member_type: VOTER last_known_addr { host: "127.28.55.67" port: 38655 } attrs { replace: true } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.509078 3384 consensus_queue.cc:237] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } }
I20240503 08:10:41.509119 3831 ts_tablet_manager.cc:1888] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:10:41.509183 3831 log.cc:1196] T 1a864e641f254544a457ddc37f4bba26 P 363ff42f68d84e7d95a885b2447673c2: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-0-root/wals/1a864e641f254544a457ddc37f4bba26
I20240503 08:10:41.509488 3704 raft_consensus.cc:1261] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Refusing update from remote peer ad402dfffae4424eb09cef218f971013: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:10:41.509655 3829 consensus_queue.cc:1035] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:41.509786 3626 raft_consensus.cc:1261] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [term 2 FOLLOWER]: Refusing update from remote peer ad402dfffae4424eb09cef218f971013: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:10:41.509924 3192 catalog_manager.cc:4830] TS 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): tablet 1a864e641f254544a457ddc37f4bba26 (table test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]) successfully deleted
I20240503 08:10:41.510042 3829 consensus_queue.cc:1035] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:10:41.510483 3793 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.510609 3704 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.511422 3189 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8b615b62f5f84fa387d8d07f6f2a1ade with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:10:41.511593 3626 raft_consensus.cc:2932] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.511608 3205 catalog_manager.cc:5484] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 reported cstate change: config changed from index 3 to 4, VOTER 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67) evicted. New cstate: current_term: 2 leader_uuid: "ad402dfffae4424eb09cef218f971013" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "363ff42f68d84e7d95a885b2447673c2" member_type: VOTER last_known_addr { host: "127.28.55.65" port: 37655 } } peers { permanent_uuid: "ad402dfffae4424eb09cef218f971013" member_type: VOTER last_known_addr { host: "127.28.55.66" port: 44797 } } peers { permanent_uuid: "cb463629df14422090d3167cc3b496ed" member_type: VOTER last_known_addr { host: "127.28.55.68" port: 39211 } attrs { promote: false } } }
I20240503 08:10:41.513025 3447 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b615b62f5f84fa387d8d07f6f2a1ade with delete_type TABLET_DATA_TOMBSTONED (TS 4dbefbfd77e7474a8a69b1c536fde51e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51372
I20240503 08:10:41.513221 3835 tablet_replica.cc:330] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e: stopping tablet replica
I20240503 08:10:41.513298 3835 raft_consensus.cc:2229] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:41.513363 3835 raft_consensus.cc:2258] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:41.513643 3835 ts_tablet_manager.cc:1875] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:10:41.514851 3835 ts_tablet_manager.cc:1888] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 08:10:41.514899 3835 log.cc:1196] T 8b615b62f5f84fa387d8d07f6f2a1ade P 4dbefbfd77e7474a8a69b1c536fde51e: Deleting WAL directory at /tmp/dist-test-task_AvDmu/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714723660874102-28893-0/minicluster-data/ts-2-root/wals/8b615b62f5f84fa387d8d07f6f2a1ade
I20240503 08:10:41.515215 3192 catalog_manager.cc:4830] TS 4dbefbfd77e7474a8a69b1c536fde51e (127.28.55.67:38655): tablet 8b615b62f5f84fa387d8d07f6f2a1ade (table test-workload [id=2da1024cb70c41f6a50aa8fd243e5297]) successfully deleted
W20240503 08:10:42.512070 3253 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 363ff42f68d84e7d95a885b2447673c2 not available for placement
I20240503 08:10:44.092335 28893 tablet_server.cc:178] TabletServer@127.28.55.65:37655 shutting down...
I20240503 08:10:44.095271 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:44.095412 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:44.095479 28893 raft_consensus.cc:2229] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:44.095539 28893 raft_consensus.cc:2258] T 6bb52224259f4909a1838a56ba8121a9 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.095703 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:44.095726 28893 raft_consensus.cc:2229] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:44.095759 28893 raft_consensus.cc:2258] T 8b615b62f5f84fa387d8d07f6f2a1ade P 363ff42f68d84e7d95a885b2447673c2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.095832 28893 tablet_replica.cc:330] stopping tablet replica
I20240503 08:10:44.095849 28893 raft_consensus.cc:2229] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:44.095887 28893 raft_consensus.cc:2258] T 85bb842493534b7cbc575a3fcf47a4c1 P 363ff42f68d84e7d95a885b2447673c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:10:44.099928 3425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.55.65:37655: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:10:44.100287 3425 consensus_peers.cc:485] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e -> Peer 363ff42f68d84e7d95a885b2447673c2 (127.28.55.65:37655): Couldn't send request to peer 363ff42f68d84e7d95a885b2447673c2. Status: Network error: Client connection negotiation failed: client connection to 127.28.55.65:37655: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:10:44.107626 28893 tablet_server.cc:194] TabletServer@127.28.55.65:37655 shutdown complete.
I20240503 08:10:44.109091 28893 tablet_server.cc:178] TabletServer@127.28.55.66:0 shutting down...
I20240503 08:10:44.111822 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:44.111979 28893 tablet_replica.cc:330] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013: stopping tablet replica
I20240503 08:10:44.112049 28893 raft_consensus.cc:2229] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:44.112128 28893 raft_consensus.cc:2258] T 1a864e641f254544a457ddc37f4bba26 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.112293 28893 tablet_replica.cc:330] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013: stopping tablet replica
I20240503 08:10:44.112344 28893 raft_consensus.cc:2229] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:10:44.112406 28893 raft_consensus.cc:2258] T 8b615b62f5f84fa387d8d07f6f2a1ade P ad402dfffae4424eb09cef218f971013 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.112522 28893 tablet_replica.cc:330] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013: stopping tablet replica
I20240503 08:10:44.112551 28893 raft_consensus.cc:2229] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:44.112586 28893 raft_consensus.cc:2258] T 6bb52224259f4909a1838a56ba8121a9 P ad402dfffae4424eb09cef218f971013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.124616 28893 tablet_server.cc:194] TabletServer@127.28.55.66:0 shutdown complete.
I20240503 08:10:44.126044 28893 tablet_server.cc:178] TabletServer@127.28.55.67:0 shutting down...
I20240503 08:10:44.129560 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:44.129676 28893 tablet_replica.cc:330] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e: stopping tablet replica
I20240503 08:10:44.129742 28893 raft_consensus.cc:2229] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:44.129820 28893 raft_consensus.cc:2258] T 1a864e641f254544a457ddc37f4bba26 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.130002 28893 tablet_replica.cc:330] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e: stopping tablet replica
I20240503 08:10:44.130041 28893 raft_consensus.cc:2229] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:44.130092 28893 raft_consensus.cc:2258] T 85bb842493534b7cbc575a3fcf47a4c1 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.130211 28893 tablet_replica.cc:330] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e: stopping tablet replica
I20240503 08:10:44.130240 28893 raft_consensus.cc:2229] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:44.130290 28893 raft_consensus.cc:2258] T 6bb52224259f4909a1838a56ba8121a9 P 4dbefbfd77e7474a8a69b1c536fde51e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.142419 28893 tablet_server.cc:194] TabletServer@127.28.55.67:0 shutdown complete.
I20240503 08:10:44.143939 28893 tablet_server.cc:178] TabletServer@127.28.55.68:0 shutting down...
I20240503 08:10:44.146551 28893 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:44.146713 28893 tablet_replica.cc:330] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed: stopping tablet replica
I20240503 08:10:44.146783 28893 raft_consensus.cc:2229] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:44.146884 28893 raft_consensus.cc:2258] T 85bb842493534b7cbc575a3fcf47a4c1 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.147060 28893 tablet_replica.cc:330] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed: stopping tablet replica
I20240503 08:10:44.147097 28893 raft_consensus.cc:2229] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:44.147148 28893 raft_consensus.cc:2258] T 8b615b62f5f84fa387d8d07f6f2a1ade P cb463629df14422090d3167cc3b496ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.147274 28893 tablet_replica.cc:330] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed: stopping tablet replica
I20240503 08:10:44.147303 28893 raft_consensus.cc:2229] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:44.147342 28893 raft_consensus.cc:2258] T 1a864e641f254544a457ddc37f4bba26 P cb463629df14422090d3167cc3b496ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.159106 28893 tablet_server.cc:194] TabletServer@127.28.55.68:0 shutdown complete.
I20240503 08:10:44.160183 28893 master.cc:537] Master@127.28.55.126:35865 shutting down...
I20240503 08:10:44.162364 28893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:44.162442 28893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:44.162469 28893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1f6309f3581d44c7b649b362a729990b: stopping tablet replica
I20240503 08:10:44.174185 28893 master.cc:552] Master@127.28.55.126:35865 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (9189 ms)
[----------] 14 tests from AutoRebalancerTest (183301 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (183301 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20240503 08:10:44.180331 28893 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:574: suppressed but not reported on 7 messages since previous log ~6 seconds ago
I20240503 08:10:44.180367 28893 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:516: suppressed but not reported on 7 messages since previous log ~6 seconds ago
I20240503 08:10:44.180379 28893 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:4573: suppressed but not reported on 6 messages since previous log ~6 seconds ago
I20240503 08:10:44.180394 28893 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5162: suppressed but not reported on 11 messages since previous log ~31 seconds ago
I20240503 08:10:44.180410 28893 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5014: suppressed but not reported on 4 messages since previous log ~77 seconds ago
I20240503 08:10:44.180431 28893 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:644: suppressed but not reported on 9 messages since previous log ~170 seconds ago
I20240503 08:10:44.180447 28893 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tablet/tablet.cc:2369: suppressed but not reported on 46 messages since previous log ~19 seconds ago