[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251026 09:09:02.346527 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:44075,127.3.14.61:45149,127.3.14.60:34921
I20251026 09:09:02.346839 3128 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 09:09:02.346980 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:02.350937 3134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:02.350953 3133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:02.351140 3128 server_base.cc:1047] running on GCE node
W20251026 09:09:02.350994 3136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:02.351428 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:02.351501 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:02.351531 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469742351531 us; error 0 us; skew 500 ppm
I20251026 09:09:02.352594 3128 webserver.cc:492] Webserver started at http://127.3.14.62:38967/ using document root <none> and password file <none>
I20251026 09:09:02.352744 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:02.352792 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:02.352880 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:02.353950 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "c46f69e0ce3741fbaf44c71c36a9f366"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.355185 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 09:09:02.355787 3141 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.355985 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.356047 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "c46f69e0ce3741fbaf44c71c36a9f366"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.356109 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:02.365473 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:02.365733 3128 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 09:09:02.365835 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:02.369900 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:44075
I20251026 09:09:02.369967 3193 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:44075 every 8 connection(s)
I20251026 09:09:02.370383 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:09:02.370558 3194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 09:09:02.371724 3197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:02.371810 3196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:02.371825 3199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:02.372063 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:02.372164 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:02.372197 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:02.372215 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469742372216 us; error 0 us; skew 500 ppm
I20251026 09:09:02.372846 3128 webserver.cc:492] Webserver started at http://127.3.14.61:41517/ using document root <none> and password file <none>
I20251026 09:09:02.372937 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:02.372982 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:02.373031 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:02.373260 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root/instance:
uuid: "aecc5d1036f040159046afd94258de78"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.373238 3194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:02.374367 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:02.374979 3206 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.375147 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.375195 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root
uuid: "aecc5d1036f040159046afd94258de78"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.375243 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:02.382696 3194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
W20251026 09:09:02.383077 3142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.14.61:45149: connect: Connection refused (error 111)
W20251026 09:09:02.383513 3194 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.14.61:45149: Network error: Client connection negotiation failed: client connection to 127.3.14.61:45149: connect: Connection refused (error 111)
I20251026 09:09:02.388324 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:02.388581 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:02.392230 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.61:45149
I20251026 09:09:02.392258 3259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.61:45149 every 8 connection(s)
I20251026 09:09:02.392591 3260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:02.392581 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:09:02.393508 3260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } has no permanent_uuid. Determining permanent_uuid...
W20251026 09:09:02.393914 3262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:02.393977 3263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:02.394418 3265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:02.394583 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:02.394717 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:02.394752 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:02.394779 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469742394779 us; error 0 us; skew 500 ppm
I20251026 09:09:02.395480 3128 webserver.cc:492] Webserver started at http://127.3.14.60:33767/ using document root <none> and password file <none>
I20251026 09:09:02.395565 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:02.395607 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:02.395654 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:02.395987 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root/instance:
uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.397233 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 09:09:02.397544 3260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:02.397796 3271 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.397914 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.397961 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root
uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.398003 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:02.402199 3260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } has no permanent_uuid. Determining permanent_uuid...
W20251026 09:09:02.402864 3260 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.14.60:34921: Network error: Client connection negotiation failed: client connection to 127.3.14.60:34921: connect: Connection refused (error 111)
I20251026 09:09:02.417075 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:02.417369 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:02.420971 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.60:34921
I20251026 09:09:02.421025 3325 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.60:34921 every 8 connection(s)
I20251026 09:09:02.421212 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:09:02.421342 3326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:02.421911 3326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:02.424919 3326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:02.427392 3326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:02.430912 3326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77: Bootstrap starting.
I20251026 09:09:02.431823 3326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:02.432087 3326 log.cc:826] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77: Log is configured to *not* fsync() on all Append() calls
I20251026 09:09:02.432705 3326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77: No bootstrap required, opened a new log
I20251026 09:09:02.432663 3194 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } attempt: 1
I20251026 09:09:02.433905 3326 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } }
I20251026 09:09:02.434005 3326 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:02.434033 3326 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb133209d8f4c8e8a6e0fa6870e0e77, State: Initialized, Role: FOLLOWER
I20251026 09:09:02.434126 3326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } }
I20251026 09:09:02.434381 3331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.434440 3326 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:02.434484 3331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:02.435559 3194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } has no permanent_uuid. Determining permanent_uuid...
W20251026 09:09:02.436460 3342 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 09:09:02.436504 3342 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 09:09:02.438928 3194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366: Bootstrap starting.
I20251026 09:09:02.439536 3194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:02.440147 3194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366: No bootstrap required, opened a new log
I20251026 09:09:02.440312 3194 raft_consensus.cc:359] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } }
I20251026 09:09:02.440366 3194 raft_consensus.cc:385] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:02.440380 3194 raft_consensus.cc:740] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46f69e0ce3741fbaf44c71c36a9f366, State: Initialized, Role: FOLLOWER
I20251026 09:09:02.440420 3194 consensus_queue.cc:260] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } }
I20251026 09:09:02.440609 3194 sys_catalog.cc:565] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:02.440611 3345 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.440733 3345 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:02.441910 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20251026 09:09:02.442049 3356 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 09:09:02.442099 3356 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 09:09:02.455029 3260 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } attempt: 1
I20251026 09:09:02.458839 3260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78: Bootstrap starting.
I20251026 09:09:02.459270 3260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:02.460249 3260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78: No bootstrap required, opened a new log
I20251026 09:09:02.460434 3260 raft_consensus.cc:359] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } }
I20251026 09:09:02.460497 3260 raft_consensus.cc:385] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:02.460511 3260 raft_consensus.cc:740] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecc5d1036f040159046afd94258de78, State: Initialized, Role: FOLLOWER
I20251026 09:09:02.460551 3260 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } }
I20251026 09:09:02.460774 3260 sys_catalog.cc:565] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:02.460784 3358 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.460919 3358 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:02.462141 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20251026 09:09:02.462312 3369 catalog_manager.cc:1568] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 09:09:02.462356 3369 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 09:09:02.462481 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:02.463915 3370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:02.463954 3371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:02.464010 3373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:02.464241 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:02.464362 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:02.464397 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:02.464424 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469742464424 us; error 0 us; skew 500 ppm
I20251026 09:09:02.465097 3128 webserver.cc:492] Webserver started at http://127.3.14.1:34735/ using document root <none> and password file <none>
I20251026 09:09:02.465184 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:02.465230 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:02.465281 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:02.465601 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "f008c49cdec14ee6b380fd55aea75aee"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.466660 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:02.467213 3378 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.467366 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.467422 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "f008c49cdec14ee6b380fd55aea75aee"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.467487 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:02.474776 3345 raft_consensus.cc:493] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:02.474885 3345 raft_consensus.cc:515] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } }
I20251026 09:09:02.475193 3345 leader_election.cc:290] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aecc5d1036f040159046afd94258de78 (127.3.14.61:45149), 2fb133209d8f4c8e8a6e0fa6870e0e77 (127.3.14.60:34921)
I20251026 09:09:02.475474 3300 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" is_pre_election: true
I20251026 09:09:02.475474 3232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aecc5d1036f040159046afd94258de78" is_pre_election: true
I20251026 09:09:02.475611 3300 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c46f69e0ce3741fbaf44c71c36a9f366 in term 0.
I20251026 09:09:02.475633 3232 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c46f69e0ce3741fbaf44c71c36a9f366 in term 0.
I20251026 09:09:02.475801 3144 leader_election.cc:304] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb133209d8f4c8e8a6e0fa6870e0e77, c46f69e0ce3741fbaf44c71c36a9f366; no voters:
I20251026 09:09:02.475900 3345 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:02.475937 3345 raft_consensus.cc:493] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:02.475962 3345 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:02.476691 3345 raft_consensus.cc:515] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } }
I20251026 09:09:02.476795 3345 leader_election.cc:290] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [CANDIDATE]: Term 1 election: Requested vote from peers aecc5d1036f040159046afd94258de78 (127.3.14.61:45149), 2fb133209d8f4c8e8a6e0fa6870e0e77 (127.3.14.60:34921)
I20251026 09:09:02.477007 3232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aecc5d1036f040159046afd94258de78"
I20251026 09:09:02.477072 3300 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77"
I20251026 09:09:02.477130 3300 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:02.477073 3232 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:02.477926 3300 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c46f69e0ce3741fbaf44c71c36a9f366 in term 1.
I20251026 09:09:02.478111 3144 leader_election.cc:304] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb133209d8f4c8e8a6e0fa6870e0e77, c46f69e0ce3741fbaf44c71c36a9f366; no voters:
I20251026 09:09:02.478161 3232 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c46f69e0ce3741fbaf44c71c36a9f366 in term 1.
I20251026 09:09:02.478277 3345 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:02.478385 3345 raft_consensus.cc:697] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 1 LEADER]: Becoming Leader. State: Replica: c46f69e0ce3741fbaf44c71c36a9f366, State: Running, Role: LEADER
I20251026 09:09:02.478464 3345 consensus_queue.cc:237] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } }
I20251026 09:09:02.478981 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c46f69e0ce3741fbaf44c71c36a9f366. Latest consensus state: current_term: 1 leader_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.478991 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:02.479158 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:02.479439 3385 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:09:02.479475 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:02.479647 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:02.479662 3385 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:09:02.479990 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:02.480036 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.480064 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:02.480083 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.481196 3300 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [term 1 FOLLOWER]: Refusing update from remote peer c46f69e0ce3741fbaf44c71c36a9f366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:02.481508 3232 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [term 1 FOLLOWER]: Refusing update from remote peer c46f69e0ce3741fbaf44c71c36a9f366: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:02.481514 3383 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:02.481703 3345 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:02.482340 3358 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c46f69e0ce3741fbaf44c71c36a9f366. Latest consensus state: current_term: 1 leader_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.482426 3358 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:02.482348 3331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c46f69e0ce3741fbaf44c71c36a9f366. Latest consensus state: current_term: 1 leader_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.482515 3331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:02.483230 3345 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.483309 3345 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:02.483382 3345 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.483433 3345 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:02.483600 3358 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.483660 3358 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:02.483743 3331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c46f69e0ce3741fbaf44c71c36a9f366" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 44075 } } peers { permanent_uuid: "aecc5d1036f040159046afd94258de78" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 45149 } } peers { permanent_uuid: "2fb133209d8f4c8e8a6e0fa6870e0e77" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 34921 } } }
I20251026 09:09:02.483796 3331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:02.483911 3389 mvcc.cc:204] Tried to move back new op lower bound from 7214980065201156096 to 7214980065192509440. Current Snapshot: MvccSnapshot[applied={T|T < 7214980065201156096}]
I20251026 09:09:02.484491 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:40031
I20251026 09:09:02.484897 3385 catalog_manager.cc:1357] Generated new cluster ID: 13542ae923ed4e2e9405222bb07b50f0
I20251026 09:09:02.484963 3385 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:09:02.484984 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:09:02.486080 3452 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:40031 every 8 connection(s)
W20251026 09:09:02.486320 3457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:02.489140 3458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:02.489365 3128 server_base.cc:1047] running on GCE node
W20251026 09:09:02.491456 3460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:02.491673 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:02.491709 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:02.491730 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469742491730 us; error 0 us; skew 500 ppm
I20251026 09:09:02.492494 3128 webserver.cc:492] Webserver started at http://127.3.14.2:40793/ using document root <none> and password file <none>
I20251026 09:09:02.492588 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:02.492630 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:02.492672 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:02.492965 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "4d09502f3f6944838b0c06b0edd2098b"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.495271 3453 heartbeater.cc:344] Connected to a master server at 127.3.14.60:34921
I20251026 09:09:02.495416 3453 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:02.495700 3453 heartbeater.cc:507] Master 127.3.14.60:34921 requested a full tablet report, sending...
I20251026 09:09:02.496205 3290 ts_manager.cc:194] Registered new tserver with Master: f008c49cdec14ee6b380fd55aea75aee (127.3.14.1:40031)
I20251026 09:09:02.497366 3455 heartbeater.cc:344] Connected to a master server at 127.3.14.61:45149
I20251026 09:09:02.497426 3455 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:02.497519 3455 heartbeater.cc:507] Master 127.3.14.61:45149 requested a full tablet report, sending...
I20251026 09:09:02.497933 3224 ts_manager.cc:194] Registered new tserver with Master: f008c49cdec14ee6b380fd55aea75aee (127.3.14.1:40031)
I20251026 09:09:02.499796 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.001s
I20251026 09:09:02.501222 3468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.501381 3128 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 09:09:02.501420 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "4d09502f3f6944838b0c06b0edd2098b"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.501470 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:02.502247 3454 heartbeater.cc:344] Connected to a master server at 127.3.14.62:44075
I20251026 09:09:02.502297 3454 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:02.502413 3454 heartbeater.cc:507] Master 127.3.14.62:44075 requested a full tablet report, sending...
I20251026 09:09:02.502609 3158 ts_manager.cc:194] Registered new tserver with Master: f008c49cdec14ee6b380fd55aea75aee (127.3.14.1:40031)
I20251026 09:09:02.522614 3385 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:09:02.523252 3385 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:09:02.532653 3385 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366: Generated new TSK 0
I20251026 09:09:02.532898 3385 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:09:02.543383 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:02.543654 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:02.543843 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:02.544054 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:02.544093 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.544126 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:02.544152 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.548563 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:40637
I20251026 09:09:02.548624 3532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:40637 every 8 connection(s)
I20251026 09:09:02.549059 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:02.550359 3539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:02.553150 3534 heartbeater.cc:344] Connected to a master server at 127.3.14.62:44075
I20251026 09:09:02.553272 3534 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:02.553323 3533 heartbeater.cc:344] Connected to a master server at 127.3.14.60:34921
I20251026 09:09:02.553386 3533 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:02.553423 3534 heartbeater.cc:507] Master 127.3.14.62:44075 requested a full tablet report, sending...
I20251026 09:09:02.553508 3533 heartbeater.cc:507] Master 127.3.14.60:34921 requested a full tablet report, sending...
I20251026 09:09:02.553653 3158 ts_manager.cc:194] Registered new tserver with Master: 4d09502f3f6944838b0c06b0edd2098b (127.3.14.2:40637)
I20251026 09:09:02.553762 3535 heartbeater.cc:344] Connected to a master server at 127.3.14.61:45149
I20251026 09:09:02.553802 3535 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:02.553839 3290 ts_manager.cc:194] Registered new tserver with Master: 4d09502f3f6944838b0c06b0edd2098b (127.3.14.2:40637)
I20251026 09:09:02.553901 3535 heartbeater.cc:507] Master 127.3.14.61:45149 requested a full tablet report, sending...
I20251026 09:09:02.554128 3224 ts_manager.cc:194] Registered new tserver with Master: 4d09502f3f6944838b0c06b0edd2098b (127.3.14.2:40637)
I20251026 09:09:02.554585 3158 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40180
W20251026 09:09:02.556255 3540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:02.556269 3542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:02.556326 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:02.556578 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:02.556622 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:02.556650 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469742556650 us; error 0 us; skew 500 ppm
I20251026 09:09:02.557433 3128 webserver.cc:492] Webserver started at http://127.3.14.3:38789/ using document root <none> and password file <none>
I20251026 09:09:02.557535 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:02.557586 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:02.557636 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:02.557955 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "c523427bebbb42119534c07e80c341c9"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.559078 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 09:09:02.559660 3548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.559791 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.559840 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "c523427bebbb42119534c07e80c341c9"
format_stamp: "Formatted at 2025-10-26 09:09:02 on dist-test-slave-w8k0"
I20251026 09:09:02.559891 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:02.571189 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:02.571491 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:02.571656 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:02.571862 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:02.571890 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.571924 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:02.571955 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:02.575503 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:36277
I20251026 09:09:02.575620 3611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:36277 every 8 connection(s)
I20251026 09:09:02.579387 3614 heartbeater.cc:344] Connected to a master server at 127.3.14.61:45149
I20251026 09:09:02.579460 3614 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:02.579561 3614 heartbeater.cc:507] Master 127.3.14.61:45149 requested a full tablet report, sending...
I20251026 09:09:02.579836 3224 ts_manager.cc:194] Registered new tserver with Master: c523427bebbb42119534c07e80c341c9 (127.3.14.3:36277)
I20251026 09:09:02.579844 3612 heartbeater.cc:344] Connected to a master server at 127.3.14.60:34921
I20251026 09:09:02.579934 3612 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:02.580034 3612 heartbeater.cc:507] Master 127.3.14.60:34921 requested a full tablet report, sending...
I20251026 09:09:02.580260 3290 ts_manager.cc:194] Registered new tserver with Master: c523427bebbb42119534c07e80c341c9 (127.3.14.3:36277)
I20251026 09:09:02.580818 3613 heartbeater.cc:344] Connected to a master server at 127.3.14.62:44075
I20251026 09:09:02.580879 3613 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:02.580968 3613 heartbeater.cc:507] Master 127.3.14.62:44075 requested a full tablet report, sending...
I20251026 09:09:02.581175 3158 ts_manager.cc:194] Registered new tserver with Master: c523427bebbb42119534c07e80c341c9 (127.3.14.3:36277)
I20251026 09:09:02.581205 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00542533s
I20251026 09:09:02.581633 3158 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40186
I20251026 09:09:03.436851 3342 catalog_manager.cc:1269] Loaded cluster ID: 13542ae923ed4e2e9405222bb07b50f0
I20251026 09:09:03.436930 3342 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77: loading cluster ID for follower catalog manager: success
I20251026 09:09:03.437717 3342 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77: acquiring CA information for follower catalog manager: success
I20251026 09:09:03.437932 3342 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 09:09:03.462656 3369 catalog_manager.cc:1269] Loaded cluster ID: 13542ae923ed4e2e9405222bb07b50f0
I20251026 09:09:03.462723 3369 catalog_manager.cc:1562] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78: loading cluster ID for follower catalog manager: success
I20251026 09:09:03.463482 3369 catalog_manager.cc:1584] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78: acquiring CA information for follower catalog manager: success
I20251026 09:09:03.463649 3369 catalog_manager.cc:1612] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 09:09:03.503468 3158 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40166
I20251026 09:09:03.555284 3534 heartbeater.cc:499] Master 127.3.14.62:44075 was elected leader, sending a full tablet report...
I20251026 09:09:03.582175 3613 heartbeater.cc:499] Master 127.3.14.62:44075 was elected leader, sending a full tablet report...
I20251026 09:09:04.441931 3355 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:04.504078 3454 heartbeater.cc:499] Master 127.3.14.62:44075 was elected leader, sending a full tablet report...
I20251026 09:09:04.635382 3128 test_util.cc:276] Using random seed: 577330673
I20251026 09:09:04.643712 3158 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40192:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 09:09:04.651121 3576 tablet_service.cc:1505] Processing CreateTablet for tablet 00674475182745ada54b4d34c69c478c (DEFAULT_TABLE table=test-workload [id=fe63d64a5dab4eb0ad9014ef45a855d2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:09:04.651317 3403 tablet_service.cc:1505] Processing CreateTablet for tablet 26c64d618088453fb0bc9130d32c40bc (DEFAULT_TABLE table=test-workload [id=fe63d64a5dab4eb0ad9014ef45a855d2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:09:04.651473 3403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c64d618088453fb0bc9130d32c40bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:04.651512 3576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00674475182745ada54b4d34c69c478c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:04.652827 3649 tablet_bootstrap.cc:492] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9: Bootstrap starting.
I20251026 09:09:04.652851 3650 tablet_bootstrap.cc:492] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee: Bootstrap starting.
I20251026 09:09:04.653100 3497 tablet_service.cc:1505] Processing CreateTablet for tablet 1ab93b3622ad45e8916beb73dbba86a7 (DEFAULT_TABLE table=test-workload [id=fe63d64a5dab4eb0ad9014ef45a855d2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:09:04.653226 3649 tablet_bootstrap.cc:654] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:04.652889 3404 tablet_service.cc:1505] Processing CreateTablet for tablet aff3fbd74eae43f6ae1251614548bb17 (DEFAULT_TABLE table=test-workload [id=fe63d64a5dab4eb0ad9014ef45a855d2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:09:04.653335 3404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aff3fbd74eae43f6ae1251614548bb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:04.653836 3649 tablet_bootstrap.cc:492] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9: No bootstrap required, opened a new log
I20251026 09:09:04.653896 3649 ts_tablet_manager.cc:1403] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:04.654033 3649 raft_consensus.cc:359] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c523427bebbb42119534c07e80c341c9" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 36277 } }
I20251026 09:09:04.654102 3649 raft_consensus.cc:385] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:04.654122 3649 raft_consensus.cc:740] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c523427bebbb42119534c07e80c341c9, State: Initialized, Role: FOLLOWER
I20251026 09:09:04.654186 3649 consensus_queue.cc:260] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c523427bebbb42119534c07e80c341c9" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 36277 } }
I20251026 09:09:04.654229 3649 raft_consensus.cc:399] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:04.654251 3649 raft_consensus.cc:493] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:04.654277 3649 raft_consensus.cc:3060] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:04.653214 3497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab93b3622ad45e8916beb73dbba86a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:04.653218 3650 tablet_bootstrap.cc:654] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:04.654928 3649 raft_consensus.cc:515] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c523427bebbb42119534c07e80c341c9" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 36277 } }
I20251026 09:09:04.655002 3649 leader_election.cc:304] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c523427bebbb42119534c07e80c341c9; no voters:
I20251026 09:09:04.655220 3649 leader_election.cc:290] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:04.655244 3650 tablet_bootstrap.cc:492] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee: No bootstrap required, opened a new log
I20251026 09:09:04.655287 3650 ts_tablet_manager.cc:1403] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:09:04.655400 3649 ts_tablet_manager.cc:1434] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 09:09:04.655439 3650 raft_consensus.cc:359] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } }
I20251026 09:09:04.655494 3650 raft_consensus.cc:385] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:04.655515 3650 raft_consensus.cc:740] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f008c49cdec14ee6b380fd55aea75aee, State: Initialized, Role: FOLLOWER
I20251026 09:09:04.655555 3650 consensus_queue.cc:260] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } }
I20251026 09:09:04.655586 3650 raft_consensus.cc:399] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:04.655603 3650 raft_consensus.cc:493] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:04.655622 3650 raft_consensus.cc:3060] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:04.655928 3655 tablet_bootstrap.cc:492] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b: Bootstrap starting.
I20251026 09:09:04.656147 3650 raft_consensus.cc:515] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } }
I20251026 09:09:04.656158 3653 raft_consensus.cc:2804] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:04.656211 3650 leader_election.cc:304] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f008c49cdec14ee6b380fd55aea75aee; no voters:
I20251026 09:09:04.656260 3653 raft_consensus.cc:697] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 1 LEADER]: Becoming Leader. State: Replica: c523427bebbb42119534c07e80c341c9, State: Running, Role: LEADER
I20251026 09:09:04.656303 3653 consensus_queue.cc:237] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c523427bebbb42119534c07e80c341c9" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 36277 } }
I20251026 09:09:04.656335 3650 leader_election.cc:290] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:04.656342 3655 tablet_bootstrap.cc:654] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:04.656497 3650 ts_tablet_manager.cc:1434] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:04.656512 3657 raft_consensus.cc:2804] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:04.656597 3650 tablet_bootstrap.cc:492] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee: Bootstrap starting.
I20251026 09:09:04.656641 3657 raft_consensus.cc:697] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 1 LEADER]: Becoming Leader. State: Replica: f008c49cdec14ee6b380fd55aea75aee, State: Running, Role: LEADER
I20251026 09:09:04.656682 3657 consensus_queue.cc:237] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } }
I20251026 09:09:04.656946 3650 tablet_bootstrap.cc:654] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:04.657181 3158 catalog_manager.cc:5649] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee reported cstate change: term changed from 0 to 1, leader changed from <none> to f008c49cdec14ee6b380fd55aea75aee (127.3.14.1). New cstate: current_term: 1 leader_uuid: "f008c49cdec14ee6b380fd55aea75aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } health_report { overall_health: HEALTHY } } }
I20251026 09:09:04.657502 3655 tablet_bootstrap.cc:492] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b: No bootstrap required, opened a new log
I20251026 09:09:04.657545 3655 ts_tablet_manager.cc:1403] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:09:04.658046 3655 raft_consensus.cc:359] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d09502f3f6944838b0c06b0edd2098b" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40637 } }
I20251026 09:09:04.658111 3655 raft_consensus.cc:385] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:04.658133 3655 raft_consensus.cc:740] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d09502f3f6944838b0c06b0edd2098b, State: Initialized, Role: FOLLOWER
I20251026 09:09:04.658200 3655 consensus_queue.cc:260] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d09502f3f6944838b0c06b0edd2098b" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40637 } }
I20251026 09:09:04.658229 3655 raft_consensus.cc:399] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:04.658533 3655 raft_consensus.cc:493] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:04.658568 3655 raft_consensus.cc:3060] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:04.659487 3655 raft_consensus.cc:515] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d09502f3f6944838b0c06b0edd2098b" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40637 } }
I20251026 09:09:04.659533 3650 tablet_bootstrap.cc:492] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee: No bootstrap required, opened a new log
I20251026 09:09:04.659559 3655 leader_election.cc:304] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d09502f3f6944838b0c06b0edd2098b; no voters:
I20251026 09:09:04.659582 3650 ts_tablet_manager.cc:1403] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 09:09:04.659670 3655 leader_election.cc:290] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:04.659698 3650 raft_consensus.cc:359] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } }
I20251026 09:09:04.659754 3650 raft_consensus.cc:385] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:04.659775 3650 raft_consensus.cc:740] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f008c49cdec14ee6b380fd55aea75aee, State: Initialized, Role: FOLLOWER
I20251026 09:09:04.659802 3655 ts_tablet_manager.cc:1434] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 09:09:04.659811 3650 consensus_queue.cc:260] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } }
I20251026 09:09:04.659888 3650 raft_consensus.cc:399] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:04.659914 3650 raft_consensus.cc:493] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:04.659978 3650 raft_consensus.cc:3060] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:04.660401 3158 catalog_manager.cc:5649] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c523427bebbb42119534c07e80c341c9 (127.3.14.3). New cstate: current_term: 1 leader_uuid: "c523427bebbb42119534c07e80c341c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c523427bebbb42119534c07e80c341c9" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 36277 } health_report { overall_health: HEALTHY } } }
I20251026 09:09:04.660776 3650 raft_consensus.cc:515] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } }
I20251026 09:09:04.660851 3650 leader_election.cc:304] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f008c49cdec14ee6b380fd55aea75aee; no voters:
I20251026 09:09:04.660915 3650 leader_election.cc:290] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:04.660984 3650 ts_tablet_manager.cc:1434] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:04.661464 3658 raft_consensus.cc:2804] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:04.661528 3658 raft_consensus.cc:697] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 1 LEADER]: Becoming Leader. State: Replica: f008c49cdec14ee6b380fd55aea75aee, State: Running, Role: LEADER
I20251026 09:09:04.661645 3658 consensus_queue.cc:237] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } }
I20251026 09:09:04.662149 3158 catalog_manager.cc:5649] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee reported cstate change: term changed from 0 to 1, leader changed from <none> to f008c49cdec14ee6b380fd55aea75aee (127.3.14.1). New cstate: current_term: 1 leader_uuid: "f008c49cdec14ee6b380fd55aea75aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f008c49cdec14ee6b380fd55aea75aee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 40031 } health_report { overall_health: HEALTHY } } }
I20251026 09:09:04.661456 3664 raft_consensus.cc:2804] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:04.662364 3664 raft_consensus.cc:697] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 1 LEADER]: Becoming Leader. State: Replica: 4d09502f3f6944838b0c06b0edd2098b, State: Running, Role: LEADER
I20251026 09:09:04.662813 3664 consensus_queue.cc:237] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d09502f3f6944838b0c06b0edd2098b" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40637 } }
I20251026 09:09:04.663538 3158 catalog_manager.cc:5649] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d09502f3f6944838b0c06b0edd2098b (127.3.14.2). New cstate: current_term: 1 leader_uuid: "4d09502f3f6944838b0c06b0edd2098b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d09502f3f6944838b0c06b0edd2098b" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40637 } health_report { overall_health: HEALTHY } } }
I20251026 09:09:04.668591 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:09:04.672016 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:04.672216 3128 tablet_replica.cc:333] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee: stopping tablet replica
I20251026 09:09:04.672304 3128 raft_consensus.cc:2243] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:04.672358 3128 raft_consensus.cc:2272] T aff3fbd74eae43f6ae1251614548bb17 P f008c49cdec14ee6b380fd55aea75aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:04.672684 3128 tablet_replica.cc:333] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee: stopping tablet replica
I20251026 09:09:04.672725 3128 raft_consensus.cc:2243] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:04.672748 3128 raft_consensus.cc:2272] T 26c64d618088453fb0bc9130d32c40bc P f008c49cdec14ee6b380fd55aea75aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:04.684728 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:09:04.685706 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:09:04.688884 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:04.689018 3128 tablet_replica.cc:333] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b: stopping tablet replica
I20251026 09:09:04.689081 3128 raft_consensus.cc:2243] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:04.689142 3128 raft_consensus.cc:2272] T 1ab93b3622ad45e8916beb73dbba86a7 P 4d09502f3f6944838b0c06b0edd2098b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:04.701013 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:09:04.701877 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:09:04.704998 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:04.705195 3128 tablet_replica.cc:333] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9: stopping tablet replica
I20251026 09:09:04.705260 3128 raft_consensus.cc:2243] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:04.705299 3128 raft_consensus.cc:2272] T 00674475182745ada54b4d34c69c478c P c523427bebbb42119534c07e80c341c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:04.717810 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:09:04.718664 3128 master.cc:561] Master@127.3.14.62:44075 shutting down...
I20251026 09:09:04.721659 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:04.721791 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:04.721838 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P c46f69e0ce3741fbaf44c71c36a9f366: stopping tablet replica
I20251026 09:09:04.734444 3128 master.cc:583] Master@127.3.14.62:44075 shutdown complete.
I20251026 09:09:04.735713 3128 master.cc:561] Master@127.3.14.61:45149 shutting down...
I20251026 09:09:04.738618 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:04.738690 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:04.738709 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P aecc5d1036f040159046afd94258de78: stopping tablet replica
I20251026 09:09:04.750833 3128 master.cc:583] Master@127.3.14.61:45149 shutdown complete.
I20251026 09:09:04.752075 3128 master.cc:561] Master@127.3.14.60:34921 shutting down...
I20251026 09:09:04.754853 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:04.754931 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:04.754973 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2fb133209d8f4c8e8a6e0fa6870e0e77: stopping tablet replica
I20251026 09:09:04.766772 3128 master.cc:583] Master@127.3.14.60:34921 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2426 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20251026 09:09:04.772440 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:43859
I20251026 09:09:04.772614 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:04.773859 3671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:04.773859 3674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:04.774024 3672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:04.774618 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:04.774772 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:04.774814 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:04.774828 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469744774829 us; error 0 us; skew 500 ppm
I20251026 09:09:04.775540 3128 webserver.cc:492] Webserver started at http://127.3.14.62:34885/ using document root <none> and password file <none>
I20251026 09:09:04.775611 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:04.775664 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:04.775720 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:04.775979 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "9b4bbe7796384fc6927d82ba13aca11c"
format_stamp: "Formatted at 2025-10-26 09:09:04 on dist-test-slave-w8k0"
I20251026 09:09:04.776908 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:04.777576 3679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.777760 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:04.777819 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "9b4bbe7796384fc6927d82ba13aca11c"
format_stamp: "Formatted at 2025-10-26 09:09:04 on dist-test-slave-w8k0"
I20251026 09:09:04.777854 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:04.783842 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:04.783996 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:04.786298 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:43859
I20251026 09:09:04.789408 3731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:43859 every 8 connection(s)
I20251026 09:09:04.789556 3732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:04.790518 3732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c: Bootstrap starting.
I20251026 09:09:04.790791 3732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:04.791278 3732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c: No bootstrap required, opened a new log
I20251026 09:09:04.791445 3732 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4bbe7796384fc6927d82ba13aca11c" member_type: VOTER }
I20251026 09:09:04.791499 3732 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:04.791522 3732 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4bbe7796384fc6927d82ba13aca11c, State: Initialized, Role: FOLLOWER
I20251026 09:09:04.791576 3732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4bbe7796384fc6927d82ba13aca11c" member_type: VOTER }
I20251026 09:09:04.791610 3732 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:04.791635 3732 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:04.791669 3732 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:04.792133 3732 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4bbe7796384fc6927d82ba13aca11c" member_type: VOTER }
I20251026 09:09:04.792188 3732 leader_election.cc:304] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b4bbe7796384fc6927d82ba13aca11c; no voters:
I20251026 09:09:04.792331 3732 leader_election.cc:290] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:04.792409 3735 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:04.792501 3732 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:04.792579 3735 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 1 LEADER]: Becoming Leader. State: Replica: 9b4bbe7796384fc6927d82ba13aca11c, State: Running, Role: LEADER
I20251026 09:09:04.792642 3735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4bbe7796384fc6927d82ba13aca11c" member_type: VOTER }
I20251026 09:09:04.792826 3735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b4bbe7796384fc6927d82ba13aca11c. Latest consensus state: current_term: 1 leader_uuid: "9b4bbe7796384fc6927d82ba13aca11c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4bbe7796384fc6927d82ba13aca11c" member_type: VOTER } }
I20251026 09:09:04.792888 3735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:04.792852 3736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b4bbe7796384fc6927d82ba13aca11c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4bbe7796384fc6927d82ba13aca11c" member_type: VOTER } }
I20251026 09:09:04.792979 3736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:04.793045 3740 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:09:04.793177 3740 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:09:04.793784 3740 catalog_manager.cc:1357] Generated new cluster ID: 4b4ba6c3336441db83cd6265d324713d
I20251026 09:09:04.793845 3740 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:09:04.793815 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:09:04.811674 3740 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:09:04.812075 3740 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:09:04.822979 3740 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c: Generated new TSK 0
I20251026 09:09:04.823071 3740 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:09:04.825706 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:04.827100 3753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:04.827171 3754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:04.827371 3756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:04.827386 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:04.827549 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:04.827585 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:04.827598 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469744827599 us; error 0 us; skew 500 ppm
I20251026 09:09:04.828176 3128 webserver.cc:492] Webserver started at http://127.3.14.1:37869/ using document root <none> and password file <none>
I20251026 09:09:04.828258 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:04.828306 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:04.828363 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:04.828655 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "c05e228b3359459292c6e388a4f27222"
format_stamp: "Formatted at 2025-10-26 09:09:04 on dist-test-slave-w8k0"
I20251026 09:09:04.829583 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:04.830112 3761 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.830219 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.830273 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "c05e228b3359459292c6e388a4f27222"
format_stamp: "Formatted at 2025-10-26 09:09:04 on dist-test-slave-w8k0"
I20251026 09:09:04.830323 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:04.839403 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:04.839639 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:04.839787 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:04.840020 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:04.840060 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.840095 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:04.840131 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.843072 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:35353
I20251026 09:09:04.843111 3824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:35353 every 8 connection(s)
I20251026 09:09:04.843518 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:04.844771 3829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:04.844811 3830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:04.844906 3832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:04.845137 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:04.845273 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:04.845310 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:04.845335 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469744845334 us; error 0 us; skew 500 ppm
I20251026 09:09:04.845919 3128 webserver.cc:492] Webserver started at http://127.3.14.2:41713/ using document root <none> and password file <none>
I20251026 09:09:04.846016 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:04.846065 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:04.846134 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:04.846415 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "99a9020a509a4b69baa2c959c4e37450"
format_stamp: "Formatted at 2025-10-26 09:09:04 on dist-test-slave-w8k0"
I20251026 09:09:04.847476 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:04.847591 3825 heartbeater.cc:344] Connected to a master server at 127.3.14.62:43859
I20251026 09:09:04.847661 3825 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:04.847776 3825 heartbeater.cc:507] Master 127.3.14.62:43859 requested a full tablet report, sending...
I20251026 09:09:04.848021 3837 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.848032 3696 ts_manager.cc:194] Registered new tserver with Master: c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353)
I20251026 09:09:04.848145 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:04.848186 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "99a9020a509a4b69baa2c959c4e37450"
format_stamp: "Formatted at 2025-10-26 09:09:04 on dist-test-slave-w8k0"
I20251026 09:09:04.848227 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:04.848918 3696 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60024
I20251026 09:09:04.860214 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:04.860455 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:04.860594 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:04.860800 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:04.860833 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.860857 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:04.860885 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.863912 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:34795
I20251026 09:09:04.864003 3900 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:34795 every 8 connection(s)
I20251026 09:09:04.864352 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:04.865839 3907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:04.865952 3128 server_base.cc:1047] running on GCE node
W20251026 09:09:04.865892 3905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:04.865803 3904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:04.866219 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:04.866269 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:04.866291 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469744866291 us; error 0 us; skew 500 ppm
I20251026 09:09:04.866968 3128 webserver.cc:492] Webserver started at http://127.3.14.3:42335/ using document root <none> and password file <none>
I20251026 09:09:04.867061 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:04.867118 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:04.867172 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:04.867525 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "14039bc98984406c9abec3ee17ad2879"
format_stamp: "Formatted at 2025-10-26 09:09:04 on dist-test-slave-w8k0"
I20251026 09:09:04.867691 3901 heartbeater.cc:344] Connected to a master server at 127.3.14.62:43859
I20251026 09:09:04.867754 3901 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:04.867851 3901 heartbeater.cc:507] Master 127.3.14.62:43859 requested a full tablet report, sending...
I20251026 09:09:04.868098 3696 ts_manager.cc:194] Registered new tserver with Master: 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795)
I20251026 09:09:04.868582 3696 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60036
I20251026 09:09:04.868673 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:04.869222 3912 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.869385 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 09:09:04.869446 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "14039bc98984406c9abec3ee17ad2879"
format_stamp: "Formatted at 2025-10-26 09:09:04 on dist-test-slave-w8k0"
I20251026 09:09:04.869480 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:04.884084 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:04.884323 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:04.884466 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:04.884662 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:04.884692 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.884716 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:04.884743 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:04.887998 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:42695
I20251026 09:09:04.888059 3975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:42695 every 8 connection(s)
I20251026 09:09:04.891249 3976 heartbeater.cc:344] Connected to a master server at 127.3.14.62:43859
I20251026 09:09:04.891314 3976 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:04.891448 3976 heartbeater.cc:507] Master 127.3.14.62:43859 requested a full tablet report, sending...
I20251026 09:09:04.891665 3696 ts_manager.cc:194] Registered new tserver with Master: 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:04.891978 3696 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60048
I20251026 09:09:04.892604 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004429675s
I20251026 09:09:05.849447 3825 heartbeater.cc:499] Master 127.3.14.62:43859 was elected leader, sending a full tablet report...
I20251026 09:09:05.869128 3901 heartbeater.cc:499] Master 127.3.14.62:43859 was elected leader, sending a full tablet report...
I20251026 09:09:05.892508 3976 heartbeater.cc:499] Master 127.3.14.62:43859 was elected leader, sending a full tablet report...
I20251026 09:09:05.918004 3128 test_util.cc:276] Using random seed: 578613295
I20251026 09:09:05.923550 3696 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60062:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:09:05.923727 3696 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:09:05.929595 3865 tablet_service.cc:1505] Processing CreateTablet for tablet 0c232f8206f341968d7df85729f7eb38 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION VALUES < 268435455
I20251026 09:09:05.929667 3862 tablet_service.cc:1505] Processing CreateTablet for tablet 62e4d046541948508a8dda9af5fbc6cb (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20251026 09:09:05.929710 3864 tablet_service.cc:1505] Processing CreateTablet for tablet 49c5577cdaab4885b395eab943d7ae8e (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20251026 09:09:05.929756 3865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c232f8206f341968d7df85729f7eb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.929881 3864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c5577cdaab4885b395eab943d7ae8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.929914 3939 tablet_service.cc:1505] Processing CreateTablet for tablet 49c5577cdaab4885b395eab943d7ae8e (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20251026 09:09:05.929976 3860 tablet_service.cc:1505] Processing CreateTablet for tablet 50309fd6edc54037b485bcf80290eb89 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20251026 09:09:05.930008 3939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c5577cdaab4885b395eab943d7ae8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.930047 3860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50309fd6edc54037b485bcf80290eb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.930233 3859 tablet_service.cc:1505] Processing CreateTablet for tablet dfc16b04f6174abdb51edf22f4abf664 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20251026 09:09:05.930313 3859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc16b04f6174abdb51edf22f4abf664. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.931284 3862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e4d046541948508a8dda9af5fbc6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.931630 3938 tablet_service.cc:1505] Processing CreateTablet for tablet e360a06882f64e0d92a723919a2cec55 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20251026 09:09:05.931713 3936 tablet_service.cc:1505] Processing CreateTablet for tablet 9e8129b4c259448287c82d541517f061 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20251026 09:09:05.931767 3938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e360a06882f64e0d92a723919a2cec55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.931830 3935 tablet_service.cc:1505] Processing CreateTablet for tablet 50309fd6edc54037b485bcf80290eb89 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20251026 09:09:05.929926 3861 tablet_service.cc:1505] Processing CreateTablet for tablet 9e8129b4c259448287c82d541517f061 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20251026 09:09:05.931928 3935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50309fd6edc54037b485bcf80290eb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.931952 3861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e8129b4c259448287c82d541517f061. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.931679 3937 tablet_service.cc:1505] Processing CreateTablet for tablet 62e4d046541948508a8dda9af5fbc6cb (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20251026 09:09:05.932986 3937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e4d046541948508a8dda9af5fbc6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.931625 3858 tablet_service.cc:1505] Processing CreateTablet for tablet 1f3bfdafccc4459ea2a3af896722f8d2 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20251026 09:09:05.933219 3934 tablet_service.cc:1505] Processing CreateTablet for tablet dfc16b04f6174abdb51edf22f4abf664 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20251026 09:09:05.933322 3858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f3bfdafccc4459ea2a3af896722f8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.933260 3933 tablet_service.cc:1505] Processing CreateTablet for tablet 1f3bfdafccc4459ea2a3af896722f8d2 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20251026 09:09:05.931636 3997 tablet_bootstrap.cc:492] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879: Bootstrap starting.
I20251026 09:09:05.929872 3940 tablet_service.cc:1505] Processing CreateTablet for tablet 0c232f8206f341968d7df85729f7eb38 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION VALUES < 268435455
I20251026 09:09:05.929840 3863 tablet_service.cc:1505] Processing CreateTablet for tablet e360a06882f64e0d92a723919a2cec55 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20251026 09:09:05.934412 3863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e360a06882f64e0d92a723919a2cec55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.933324 3936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e8129b4c259448287c82d541517f061. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.935704 3940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c232f8206f341968d7df85729f7eb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.935797 3933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f3bfdafccc4459ea2a3af896722f8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.935913 3934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc16b04f6174abdb51edf22f4abf664. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.937422 3997 tablet_bootstrap.cc:654] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.938042 3997 tablet_bootstrap.cc:492] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879: No bootstrap required, opened a new log
I20251026 09:09:05.938103 3997 ts_tablet_manager.cc:1403] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20251026 09:09:05.938287 3997 raft_consensus.cc:359] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.938370 3997 raft_consensus.cc:385] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.938417 3997 raft_consensus.cc:740] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.938483 3997 consensus_queue.cc:260] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.938578 3996 tablet_bootstrap.cc:492] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450: Bootstrap starting.
I20251026 09:09:05.938653 3997 ts_tablet_manager.cc:1434] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.938694 3789 tablet_service.cc:1505] Processing CreateTablet for tablet 0c232f8206f341968d7df85729f7eb38 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION VALUES < 268435455
I20251026 09:09:05.938808 3789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c232f8206f341968d7df85729f7eb38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.938822 3997 tablet_bootstrap.cc:492] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879: Bootstrap starting.
I20251026 09:09:05.939254 3996 tablet_bootstrap.cc:654] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.939250 3997 tablet_bootstrap.cc:654] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.940083 3996 tablet_bootstrap.cc:492] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450: No bootstrap required, opened a new log
I20251026 09:09:05.940150 3996 ts_tablet_manager.cc:1403] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:09:05.940186 3788 tablet_service.cc:1505] Processing CreateTablet for tablet 49c5577cdaab4885b395eab943d7ae8e (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20251026 09:09:05.940265 4002 tablet_bootstrap.cc:492] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222: Bootstrap starting.
I20251026 09:09:05.940299 3788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c5577cdaab4885b395eab943d7ae8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.940332 3996 raft_consensus.cc:359] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.940402 3996 raft_consensus.cc:385] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.940430 3996 raft_consensus.cc:740] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.940478 3996 consensus_queue.cc:260] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.940636 3996 ts_tablet_manager.cc:1434] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:09:05.940729 3996 tablet_bootstrap.cc:492] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450: Bootstrap starting.
I20251026 09:09:05.940831 4002 tablet_bootstrap.cc:654] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.941113 3996 tablet_bootstrap.cc:654] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.941514 3787 tablet_service.cc:1505] Processing CreateTablet for tablet e360a06882f64e0d92a723919a2cec55 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20251026 09:09:05.941618 3787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e360a06882f64e0d92a723919a2cec55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.941602 3997 tablet_bootstrap.cc:492] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879: No bootstrap required, opened a new log
I20251026 09:09:05.941694 3997 ts_tablet_manager.cc:1403] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 09:09:05.941738 3996 tablet_bootstrap.cc:492] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450: No bootstrap required, opened a new log
I20251026 09:09:05.941785 3996 ts_tablet_manager.cc:1403] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:05.941847 3997 raft_consensus.cc:359] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.941915 3997 raft_consensus.cc:385] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.941936 3997 raft_consensus.cc:740] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.941987 3997 consensus_queue.cc:260] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.942072 3997 ts_tablet_manager.cc:1434] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.942131 3997 tablet_bootstrap.cc:492] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879: Bootstrap starting.
I20251026 09:09:05.941936 3996 raft_consensus.cc:359] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.942169 3996 raft_consensus.cc:385] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.942193 3996 raft_consensus.cc:740] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.942250 3996 consensus_queue.cc:260] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.942343 3996 ts_tablet_manager.cc:1434] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:05.942404 3996 tablet_bootstrap.cc:492] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450: Bootstrap starting.
I20251026 09:09:05.942543 3997 tablet_bootstrap.cc:654] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.942798 3996 tablet_bootstrap.cc:654] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.942948 3786 tablet_service.cc:1505] Processing CreateTablet for tablet 62e4d046541948508a8dda9af5fbc6cb (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20251026 09:09:05.943015 3786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e4d046541948508a8dda9af5fbc6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.943485 3996 tablet_bootstrap.cc:492] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450: No bootstrap required, opened a new log
I20251026 09:09:05.943534 3996 ts_tablet_manager.cc:1403] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:05.943678 3996 raft_consensus.cc:359] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.943740 3996 raft_consensus.cc:385] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.943766 3996 raft_consensus.cc:740] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.943835 3996 consensus_queue.cc:260] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.943986 3996 ts_tablet_manager.cc:1434] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:09:05.944061 3996 tablet_bootstrap.cc:492] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450: Bootstrap starting.
I20251026 09:09:05.944254 3785 tablet_service.cc:1505] Processing CreateTablet for tablet 9e8129b4c259448287c82d541517f061 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20251026 09:09:05.944353 3785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e8129b4c259448287c82d541517f061. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.944482 3996 tablet_bootstrap.cc:654] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.945103 3996 tablet_bootstrap.cc:492] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450: No bootstrap required, opened a new log
I20251026 09:09:05.945175 3996 ts_tablet_manager.cc:1403] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:05.945322 3996 raft_consensus.cc:359] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.945370 3996 raft_consensus.cc:385] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.945384 3996 raft_consensus.cc:740] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.945436 3996 consensus_queue.cc:260] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.945539 3996 ts_tablet_manager.cc:1434] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.945596 3996 tablet_bootstrap.cc:492] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450: Bootstrap starting.
I20251026 09:09:05.945585 3784 tablet_service.cc:1505] Processing CreateTablet for tablet 50309fd6edc54037b485bcf80290eb89 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20251026 09:09:05.945690 3783 tablet_service.cc:1505] Processing CreateTablet for tablet dfc16b04f6174abdb51edf22f4abf664 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20251026 09:09:05.945737 3784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50309fd6edc54037b485bcf80290eb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.945837 3782 tablet_service.cc:1505] Processing CreateTablet for tablet 1f3bfdafccc4459ea2a3af896722f8d2 (DEFAULT_TABLE table=test-workload [id=25796c69f4e44dbd95bbe7c50ace60e7]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20251026 09:09:05.945926 3782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f3bfdafccc4459ea2a3af896722f8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.947022 3783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc16b04f6174abdb51edf22f4abf664. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:05.947067 3996 tablet_bootstrap.cc:654] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.947726 3996 tablet_bootstrap.cc:492] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450: No bootstrap required, opened a new log
I20251026 09:09:05.947773 3996 ts_tablet_manager.cc:1403] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:09:05.947894 3996 raft_consensus.cc:359] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.947942 3996 raft_consensus.cc:385] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.947966 3996 raft_consensus.cc:740] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.948038 3996 consensus_queue.cc:260] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.948132 3996 ts_tablet_manager.cc:1434] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.948328 3996 tablet_bootstrap.cc:492] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450: Bootstrap starting.
I20251026 09:09:05.948387 3997 tablet_bootstrap.cc:492] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879: No bootstrap required, opened a new log
I20251026 09:09:05.948436 3997 ts_tablet_manager.cc:1403] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20251026 09:09:05.948493 4002 tablet_bootstrap.cc:492] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222: No bootstrap required, opened a new log
I20251026 09:09:05.948529 4002 ts_tablet_manager.cc:1403] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20251026 09:09:05.948575 3997 raft_consensus.cc:359] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.948632 3997 raft_consensus.cc:385] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.948652 4002 raft_consensus.cc:359] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.948720 4002 raft_consensus.cc:385] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.948737 4002 raft_consensus.cc:740] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05e228b3359459292c6e388a4f27222, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.948652 3997 raft_consensus.cc:740] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.948793 4002 consensus_queue.cc:260] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.948812 3997 consensus_queue.cc:260] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.948832 3996 tablet_bootstrap.cc:654] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.948904 3997 ts_tablet_manager.cc:1434] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.948935 4002 ts_tablet_manager.cc:1434] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:05.948959 3997 tablet_bootstrap.cc:492] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879: Bootstrap starting.
I20251026 09:09:05.948997 4002 tablet_bootstrap.cc:492] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222: Bootstrap starting.
I20251026 09:09:05.949420 3997 tablet_bootstrap.cc:654] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.949420 4002 tablet_bootstrap.cc:654] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.950006 3997 tablet_bootstrap.cc:492] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879: No bootstrap required, opened a new log
I20251026 09:09:05.950060 3997 ts_tablet_manager.cc:1403] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.950078 4002 tablet_bootstrap.cc:492] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222: No bootstrap required, opened a new log
I20251026 09:09:05.950129 4002 ts_tablet_manager.cc:1403] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.950239 3997 raft_consensus.cc:359] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.950305 3997 raft_consensus.cc:385] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.950345 3997 raft_consensus.cc:740] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.950387 3997 consensus_queue.cc:260] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.950464 3997 ts_tablet_manager.cc:1434] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:05.950510 3997 tablet_bootstrap.cc:492] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879: Bootstrap starting.
I20251026 09:09:05.950668 3996 tablet_bootstrap.cc:492] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450: No bootstrap required, opened a new log
I20251026 09:09:05.950709 3996 ts_tablet_manager.cc:1403] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:09:05.950846 3996 raft_consensus.cc:359] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.950877 3997 tablet_bootstrap.cc:654] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.950893 3996 raft_consensus.cc:385] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.950907 3996 raft_consensus.cc:740] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.950939 3996 consensus_queue.cc:260] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.951041 3996 ts_tablet_manager.cc:1434] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.951097 3996 tablet_bootstrap.cc:492] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450: Bootstrap starting.
I20251026 09:09:05.950313 4002 raft_consensus.cc:359] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.951385 4002 raft_consensus.cc:385] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.951416 4002 raft_consensus.cc:740] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05e228b3359459292c6e388a4f27222, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.951459 4002 consensus_queue.cc:260] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.951566 4002 ts_tablet_manager.cc:1434] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.951551 3996 tablet_bootstrap.cc:654] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.951675 4002 tablet_bootstrap.cc:492] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222: Bootstrap starting.
I20251026 09:09:05.952111 4002 tablet_bootstrap.cc:654] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.952251 3996 tablet_bootstrap.cc:492] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450: No bootstrap required, opened a new log
I20251026 09:09:05.952291 3996 ts_tablet_manager.cc:1403] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.952399 3996 raft_consensus.cc:359] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.952466 3996 raft_consensus.cc:385] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.952487 3996 raft_consensus.cc:740] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.952531 3996 consensus_queue.cc:260] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.952787 3996 ts_tablet_manager.cc:1434] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:05.952859 3996 tablet_bootstrap.cc:492] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450: Bootstrap starting.
I20251026 09:09:05.952903 4002 tablet_bootstrap.cc:492] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222: No bootstrap required, opened a new log
I20251026 09:09:05.952946 4002 ts_tablet_manager.cc:1403] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.953094 4002 raft_consensus.cc:359] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.953159 4002 raft_consensus.cc:385] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.953172 3996 tablet_bootstrap.cc:654] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.953209 4002 raft_consensus.cc:740] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05e228b3359459292c6e388a4f27222, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.953286 4002 consensus_queue.cc:260] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.953373 4002 ts_tablet_manager.cc:1434] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:05.953428 4002 tablet_bootstrap.cc:492] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222: Bootstrap starting.
I20251026 09:09:05.953778 3996 tablet_bootstrap.cc:492] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450: No bootstrap required, opened a new log
I20251026 09:09:05.953821 3996 ts_tablet_manager.cc:1403] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.953831 4002 tablet_bootstrap.cc:654] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.953931 3996 raft_consensus.cc:359] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.953975 3996 raft_consensus.cc:385] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.953989 3996 raft_consensus.cc:740] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.954022 3996 consensus_queue.cc:260] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.954134 3996 ts_tablet_manager.cc:1434] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.954595 3997 tablet_bootstrap.cc:492] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879: No bootstrap required, opened a new log
I20251026 09:09:05.954648 3997 ts_tablet_manager.cc:1403] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251026 09:09:05.954694 4002 tablet_bootstrap.cc:492] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222: No bootstrap required, opened a new log
I20251026 09:09:05.954747 4002 ts_tablet_manager.cc:1403] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.954799 3997 raft_consensus.cc:359] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.954855 3997 raft_consensus.cc:385] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.954875 3997 raft_consensus.cc:740] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.954909 4002 raft_consensus.cc:359] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.954921 3997 consensus_queue.cc:260] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.954996 4002 raft_consensus.cc:385] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.955027 4002 raft_consensus.cc:740] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05e228b3359459292c6e388a4f27222, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.955049 3997 ts_tablet_manager.cc:1434] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:05.955073 4002 consensus_queue.cc:260] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.955119 3997 tablet_bootstrap.cc:492] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879: Bootstrap starting.
I20251026 09:09:05.955191 4002 ts_tablet_manager.cc:1434] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:05.955267 4002 tablet_bootstrap.cc:492] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222: Bootstrap starting.
I20251026 09:09:05.955565 3997 tablet_bootstrap.cc:654] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.955698 4002 tablet_bootstrap.cc:654] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.956182 3997 tablet_bootstrap.cc:492] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879: No bootstrap required, opened a new log
I20251026 09:09:05.956240 3997 ts_tablet_manager.cc:1403] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:05.956314 4002 tablet_bootstrap.cc:492] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222: No bootstrap required, opened a new log
I20251026 09:09:05.956368 4002 ts_tablet_manager.cc:1403] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.956391 3997 raft_consensus.cc:359] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.956463 3997 raft_consensus.cc:385] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.956489 3997 raft_consensus.cc:740] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.956499 4002 raft_consensus.cc:359] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.956539 3997 consensus_queue.cc:260] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.956570 4002 raft_consensus.cc:385] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.956602 4002 raft_consensus.cc:740] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05e228b3359459292c6e388a4f27222, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.956620 3997 ts_tablet_manager.cc:1434] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.956648 4002 consensus_queue.cc:260] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.956679 3997 tablet_bootstrap.cc:492] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879: Bootstrap starting.
I20251026 09:09:05.956735 4002 ts_tablet_manager.cc:1434] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:05.956794 4002 tablet_bootstrap.cc:492] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222: Bootstrap starting.
I20251026 09:09:05.957054 3997 tablet_bootstrap.cc:654] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.957249 4002 tablet_bootstrap.cc:654] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.957769 4002 tablet_bootstrap.cc:492] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222: No bootstrap required, opened a new log
I20251026 09:09:05.957819 4002 ts_tablet_manager.cc:1403] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:05.957970 4002 raft_consensus.cc:359] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.957973 3997 tablet_bootstrap.cc:492] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879: No bootstrap required, opened a new log
I20251026 09:09:05.958058 4002 raft_consensus.cc:385] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.958078 3997 ts_tablet_manager.cc:1403] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:05.958082 4002 raft_consensus.cc:740] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05e228b3359459292c6e388a4f27222, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.958200 4002 consensus_queue.cc:260] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.958245 3997 raft_consensus.cc:359] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.958292 4002 ts_tablet_manager.cc:1434] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.958312 3997 raft_consensus.cc:385] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.958341 3997 raft_consensus.cc:740] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.958366 4002 tablet_bootstrap.cc:492] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222: Bootstrap starting.
I20251026 09:09:05.958408 3997 consensus_queue.cc:260] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.958504 3997 ts_tablet_manager.cc:1434] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.958571 3997 tablet_bootstrap.cc:492] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879: Bootstrap starting.
I20251026 09:09:05.958793 4002 tablet_bootstrap.cc:654] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.959856 3997 tablet_bootstrap.cc:654] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.960026 4002 tablet_bootstrap.cc:492] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222: No bootstrap required, opened a new log
I20251026 09:09:05.960073 4002 ts_tablet_manager.cc:1403] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:09:05.960239 4002 raft_consensus.cc:359] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.960318 4002 raft_consensus.cc:385] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.960350 4002 raft_consensus.cc:740] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05e228b3359459292c6e388a4f27222, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.960434 4002 consensus_queue.cc:260] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.960469 3997 tablet_bootstrap.cc:492] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879: No bootstrap required, opened a new log
I20251026 09:09:05.960549 4002 ts_tablet_manager.cc:1434] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:09:05.960558 3997 ts_tablet_manager.cc:1403] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:09:05.960621 4002 tablet_bootstrap.cc:492] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222: Bootstrap starting.
I20251026 09:09:05.960727 3997 raft_consensus.cc:359] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.960803 3997 raft_consensus.cc:385] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.960824 3997 raft_consensus.cc:740] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.960897 3997 consensus_queue.cc:260] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.960992 3997 ts_tablet_manager.cc:1434] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.961122 4002 tablet_bootstrap.cc:654] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:05.961828 4002 tablet_bootstrap.cc:492] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222: No bootstrap required, opened a new log
I20251026 09:09:05.961871 4002 ts_tablet_manager.cc:1403] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:05.961974 4002 raft_consensus.cc:359] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.962015 4002 raft_consensus.cc:385] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:05.962028 4002 raft_consensus.cc:740] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05e228b3359459292c6e388a4f27222, State: Initialized, Role: FOLLOWER
I20251026 09:09:05.962064 4002 consensus_queue.cc:260] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.962158 4002 ts_tablet_manager.cc:1434] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:05.966341 4011 raft_consensus.cc:493] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:05.966403 4011 raft_consensus.cc:515] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.966542 4011 leader_election.cc:290] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:05.969740 3875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f3bfdafccc4459ea2a3af896722f8d2" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450" is_pre_election: true
I20251026 09:09:05.969818 3875 raft_consensus.cc:2468] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 0.
I20251026 09:09:05.969980 3765 leader_election.cc:304] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a9020a509a4b69baa2c959c4e37450, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:05.969980 3950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f3bfdafccc4459ea2a3af896722f8d2" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879" is_pre_election: true
I20251026 09:09:05.970080 3950 raft_consensus.cc:2468] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 0.
I20251026 09:09:05.970104 4011 raft_consensus.cc:2804] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:05.970155 4011 raft_consensus.cc:493] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:05.970188 4011 raft_consensus.cc:3060] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:05.970832 4011 raft_consensus.cc:515] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.970953 4011 leader_election.cc:290] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 election: Requested vote from peers 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:05.971050 4011 raft_consensus.cc:493] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:05.971089 4011 raft_consensus.cc:515] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.971114 3999 raft_consensus.cc:493] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:05.971114 3875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f3bfdafccc4459ea2a3af896722f8d2" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450"
I20251026 09:09:05.971180 4011 leader_election.cc:290] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:05.971212 3875 raft_consensus.cc:3060] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:05.971204 3950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f3bfdafccc4459ea2a3af896722f8d2" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879"
I20251026 09:09:05.971268 3950 raft_consensus.cc:3060] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:05.971359 3874 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62e4d046541948508a8dda9af5fbc6cb" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450" is_pre_election: true
I20251026 09:09:05.971429 3874 raft_consensus.cc:2468] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 0.
I20251026 09:09:05.971551 3765 leader_election.cc:304] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a9020a509a4b69baa2c959c4e37450, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:05.971683 4011 raft_consensus.cc:2804] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:05.971720 4011 raft_consensus.cc:493] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:05.971740 4011 raft_consensus.cc:3060] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:05.971997 3875 raft_consensus.cc:2468] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 1.
I20251026 09:09:05.972119 3950 raft_consensus.cc:2468] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 1.
I20251026 09:09:05.972141 3765 leader_election.cc:304] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a9020a509a4b69baa2c959c4e37450, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:05.971184 3999 raft_consensus.cc:515] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.972340 4011 raft_consensus.cc:515] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.972440 4011 leader_election.cc:290] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 election: Requested vote from peers 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:05.972371 4029 raft_consensus.cc:2804] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:05.972600 4029 raft_consensus.cc:697] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Becoming Leader. State: Replica: c05e228b3359459292c6e388a4f27222, State: Running, Role: LEADER
I20251026 09:09:05.972621 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62e4d046541948508a8dda9af5fbc6cb" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879"
I20251026 09:09:05.972388 3999 leader_election.cc:290] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353), 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795)
I20251026 09:09:05.972707 3949 raft_consensus.cc:3060] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:05.972769 3875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62e4d046541948508a8dda9af5fbc6cb" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450"
I20251026 09:09:05.972806 4029 consensus_queue.cc:237] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.972834 3875 raft_consensus.cc:3060] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:05.973093 3950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62e4d046541948508a8dda9af5fbc6cb" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879" is_pre_election: true
I20251026 09:09:05.973378 3949 raft_consensus.cc:2468] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 1.
I20251026 09:09:05.973482 3695 catalog_manager.cc:5649] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 reported cstate change: term changed from 0 to 1, leader changed from <none> to c05e228b3359459292c6e388a4f27222 (127.3.14.1). New cstate: current_term: 1 leader_uuid: "c05e228b3359459292c6e388a4f27222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:05.973541 3763 leader_election.cc:304] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:05.973635 3875 raft_consensus.cc:2468] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 1.
I20251026 09:09:05.973793 4011 raft_consensus.cc:2804] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:05.973886 4011 raft_consensus.cc:697] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Becoming Leader. State: Replica: c05e228b3359459292c6e388a4f27222, State: Running, Role: LEADER
I20251026 09:09:05.973923 4011 consensus_queue.cc:237] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.974575 3695 catalog_manager.cc:5649] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 reported cstate change: term changed from 0 to 1, leader changed from <none> to c05e228b3359459292c6e388a4f27222 (127.3.14.1). New cstate: current_term: 1 leader_uuid: "c05e228b3359459292c6e388a4f27222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:05.976332 3799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e8129b4c259448287c82d541517f061" candidate_uuid: "14039bc98984406c9abec3ee17ad2879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05e228b3359459292c6e388a4f27222" is_pre_election: true
I20251026 09:09:05.976421 3799 raft_consensus.cc:2468] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14039bc98984406c9abec3ee17ad2879 in term 0.
I20251026 09:09:05.976639 3914 leader_election.cc:304] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:05.976750 3875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e8129b4c259448287c82d541517f061" candidate_uuid: "14039bc98984406c9abec3ee17ad2879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450" is_pre_election: true
I20251026 09:09:05.976822 3875 raft_consensus.cc:2468] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14039bc98984406c9abec3ee17ad2879 in term 0.
I20251026 09:09:05.976749 3999 raft_consensus.cc:2804] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:05.976917 3999 raft_consensus.cc:493] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:05.976944 3999 raft_consensus.cc:3060] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:05.977494 3999 raft_consensus.cc:515] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.977603 3999 leader_election.cc:290] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [CANDIDATE]: Term 1 election: Requested vote from peers c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353), 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795)
I20251026 09:09:05.977769 3875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e8129b4c259448287c82d541517f061" candidate_uuid: "14039bc98984406c9abec3ee17ad2879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450"
I20251026 09:09:05.977792 3799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e8129b4c259448287c82d541517f061" candidate_uuid: "14039bc98984406c9abec3ee17ad2879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05e228b3359459292c6e388a4f27222"
I20251026 09:09:05.977828 3875 raft_consensus.cc:3060] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:05.977835 3799 raft_consensus.cc:3060] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:05.978273 3875 raft_consensus.cc:2468] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14039bc98984406c9abec3ee17ad2879 in term 1.
I20251026 09:09:05.978399 3799 raft_consensus.cc:2468] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14039bc98984406c9abec3ee17ad2879 in term 1.
I20251026 09:09:05.978456 3916 leader_election.cc:304] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, 99a9020a509a4b69baa2c959c4e37450; no voters:
I20251026 09:09:05.978525 3999 raft_consensus.cc:2804] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:05.978641 3999 raft_consensus.cc:697] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 1 LEADER]: Becoming Leader. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Running, Role: LEADER
I20251026 09:09:05.978688 3999 consensus_queue.cc:237] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:05.979209 3695 catalog_manager.cc:5649] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14039bc98984406c9abec3ee17ad2879 (127.3.14.3). New cstate: current_term: 1 leader_uuid: "14039bc98984406c9abec3ee17ad2879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: HEALTHY } } }
I20251026 09:09:06.008772 4029 raft_consensus.cc:493] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:06.008895 4029 raft_consensus.cc:515] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.009104 4029 leader_election.cc:290] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.009356 3875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e360a06882f64e0d92a723919a2cec55" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450" is_pre_election: true
I20251026 09:09:06.009380 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e360a06882f64e0d92a723919a2cec55" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879" is_pre_election: true
I20251026 09:09:06.009480 3875 raft_consensus.cc:2468] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 0.
I20251026 09:09:06.009490 3949 raft_consensus.cc:2468] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 0.
I20251026 09:09:06.009667 3763 leader_election.cc:304] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:06.009769 4029 raft_consensus.cc:2804] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:06.009809 4029 raft_consensus.cc:493] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:06.009840 4029 raft_consensus.cc:3060] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.010505 4029 raft_consensus.cc:515] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.010630 4029 leader_election.cc:290] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 election: Requested vote from peers 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.010807 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e360a06882f64e0d92a723919a2cec55" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879"
I20251026 09:09:06.010828 3875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e360a06882f64e0d92a723919a2cec55" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450"
I20251026 09:09:06.010916 3949 raft_consensus.cc:3060] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.010926 3875 raft_consensus.cc:3060] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.011624 3949 raft_consensus.cc:2468] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 1.
I20251026 09:09:06.011624 3875 raft_consensus.cc:2468] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 1.
I20251026 09:09:06.011886 3763 leader_election.cc:304] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:06.011996 4029 raft_consensus.cc:2804] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:06.012073 4029 raft_consensus.cc:697] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Becoming Leader. State: Replica: c05e228b3359459292c6e388a4f27222, State: Running, Role: LEADER
I20251026 09:09:06.012136 4029 consensus_queue.cc:237] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.012871 3695 catalog_manager.cc:5649] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 reported cstate change: term changed from 0 to 1, leader changed from <none> to c05e228b3359459292c6e388a4f27222 (127.3.14.1). New cstate: current_term: 1 leader_uuid: "c05e228b3359459292c6e388a4f27222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: UNKNOWN } } }
W20251026 09:09:06.094425 3826 tablet.cc:2378] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 09:09:06.128355 4003 raft_consensus.cc:493] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:06.128473 4003 raft_consensus.cc:515] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.128684 4003 leader_election.cc:290] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.131654 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50309fd6edc54037b485bcf80290eb89" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879" is_pre_election: true
I20251026 09:09:06.131738 3949 raft_consensus.cc:2468] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 0.
I20251026 09:09:06.131686 3799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50309fd6edc54037b485bcf80290eb89" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05e228b3359459292c6e388a4f27222" is_pre_election: true
I20251026 09:09:06.131780 3799 raft_consensus.cc:2468] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 0.
I20251026 09:09:06.131922 3839 leader_election.cc:304] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, 99a9020a509a4b69baa2c959c4e37450; no voters:
I20251026 09:09:06.132019 4003 raft_consensus.cc:2804] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:06.132067 4003 raft_consensus.cc:493] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:06.132093 4003 raft_consensus.cc:3060] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.132751 4003 raft_consensus.cc:515] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.132874 4003 leader_election.cc:290] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 election: Requested vote from peers c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.133026 3799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50309fd6edc54037b485bcf80290eb89" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05e228b3359459292c6e388a4f27222"
I20251026 09:09:06.133064 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50309fd6edc54037b485bcf80290eb89" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879"
I20251026 09:09:06.133085 3799 raft_consensus.cc:3060] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.133111 3949 raft_consensus.cc:3060] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.133404 4029 raft_consensus.cc:493] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:06.133500 4029 raft_consensus.cc:515] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.133673 3949 raft_consensus.cc:2468] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 1.
I20251026 09:09:06.133687 3799 raft_consensus.cc:2468] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 1.
I20251026 09:09:06.133786 4029 leader_election.cc:290] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.133831 3839 leader_election.cc:304] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, 99a9020a509a4b69baa2c959c4e37450; no voters:
I20251026 09:09:06.133905 4003 raft_consensus.cc:2804] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:06.133958 4003 raft_consensus.cc:697] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 1 LEADER]: Becoming Leader. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Running, Role: LEADER
I20251026 09:09:06.133980 3875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49c5577cdaab4885b395eab943d7ae8e" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450" is_pre_election: true
I20251026 09:09:06.133980 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49c5577cdaab4885b395eab943d7ae8e" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879" is_pre_election: true
I20251026 09:09:06.134083 3875 raft_consensus.cc:2468] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 0.
I20251026 09:09:06.134104 3949 raft_consensus.cc:2468] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 0.
I20251026 09:09:06.134011 4003 consensus_queue.cc:237] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.134223 3765 leader_election.cc:304] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a9020a509a4b69baa2c959c4e37450, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:06.134342 4029 raft_consensus.cc:2804] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:06.134378 4029 raft_consensus.cc:493] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:06.134394 4029 raft_consensus.cc:3060] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.134788 3695 catalog_manager.cc:5649] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "99a9020a509a4b69baa2c959c4e37450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:06.134955 4029 raft_consensus.cc:515] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.135234 4029 leader_election.cc:290] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 election: Requested vote from peers 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.135210 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49c5577cdaab4885b395eab943d7ae8e" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879"
I20251026 09:09:06.135502 3949 raft_consensus.cc:3060] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.135957 3949 raft_consensus.cc:2468] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 1.
I20251026 09:09:06.135226 3875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49c5577cdaab4885b395eab943d7ae8e" candidate_uuid: "c05e228b3359459292c6e388a4f27222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99a9020a509a4b69baa2c959c4e37450"
I20251026 09:09:06.136089 3875 raft_consensus.cc:3060] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.136137 3763 leader_election.cc:304] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:06.136245 4029 raft_consensus.cc:2804] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:06.136303 4029 raft_consensus.cc:697] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Becoming Leader. State: Replica: c05e228b3359459292c6e388a4f27222, State: Running, Role: LEADER
I20251026 09:09:06.136338 4029 consensus_queue.cc:237] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.136698 3875 raft_consensus.cc:2468] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05e228b3359459292c6e388a4f27222 in term 1.
I20251026 09:09:06.136898 3695 catalog_manager.cc:5649] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 reported cstate change: term changed from 0 to 1, leader changed from <none> to c05e228b3359459292c6e388a4f27222 (127.3.14.1). New cstate: current_term: 1 leader_uuid: "c05e228b3359459292c6e388a4f27222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:06.137498 4003 raft_consensus.cc:493] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:06.137558 4003 raft_consensus.cc:515] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.137668 4003 leader_election.cc:290] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.137859 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfc16b04f6174abdb51edf22f4abf664" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879" is_pre_election: true
I20251026 09:09:06.137859 3799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfc16b04f6174abdb51edf22f4abf664" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05e228b3359459292c6e388a4f27222" is_pre_election: true
I20251026 09:09:06.137954 3949 raft_consensus.cc:2468] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 0.
I20251026 09:09:06.137974 3799 raft_consensus.cc:2468] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 0.
I20251026 09:09:06.138062 3839 leader_election.cc:304] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, 99a9020a509a4b69baa2c959c4e37450; no voters:
I20251026 09:09:06.138136 4003 raft_consensus.cc:2804] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:06.138199 4003 raft_consensus.cc:493] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:06.138219 4003 raft_consensus.cc:3060] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.138662 4003 raft_consensus.cc:515] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.138749 4003 leader_election.cc:290] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 election: Requested vote from peers c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.138932 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfc16b04f6174abdb51edf22f4abf664" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879"
I20251026 09:09:06.138911 3799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfc16b04f6174abdb51edf22f4abf664" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05e228b3359459292c6e388a4f27222"
I20251026 09:09:06.139006 3949 raft_consensus.cc:3060] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.139036 3799 raft_consensus.cc:3060] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.139508 3949 raft_consensus.cc:2468] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 1.
I20251026 09:09:06.139627 3799 raft_consensus.cc:2468] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 1.
I20251026 09:09:06.139655 3839 leader_election.cc:304] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, 99a9020a509a4b69baa2c959c4e37450; no voters:
I20251026 09:09:06.139726 4003 raft_consensus.cc:2804] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:06.139784 4003 raft_consensus.cc:697] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 1 LEADER]: Becoming Leader. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Running, Role: LEADER
I20251026 09:09:06.139837 4003 consensus_queue.cc:237] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.140339 3695 catalog_manager.cc:5649] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "99a9020a509a4b69baa2c959c4e37450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:06.231501 4003 raft_consensus.cc:493] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:06.231627 4003 raft_consensus.cc:515] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.231817 4003 leader_election.cc:290] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.232057 3799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c232f8206f341968d7df85729f7eb38" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05e228b3359459292c6e388a4f27222" is_pre_election: true
I20251026 09:09:06.232064 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c232f8206f341968d7df85729f7eb38" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879" is_pre_election: true
I20251026 09:09:06.232131 3799 raft_consensus.cc:2468] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 0.
I20251026 09:09:06.232179 3949 raft_consensus.cc:2468] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 0.
I20251026 09:09:06.232314 3839 leader_election.cc:304] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99a9020a509a4b69baa2c959c4e37450, c05e228b3359459292c6e388a4f27222; no voters:
I20251026 09:09:06.232450 4003 raft_consensus.cc:2804] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:06.232486 4003 raft_consensus.cc:493] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:06.232501 4003 raft_consensus.cc:3060] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.233062 4003 raft_consensus.cc:515] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.233163 4003 leader_election.cc:290] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 election: Requested vote from peers c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353), 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:06.233392 3799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c232f8206f341968d7df85729f7eb38" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05e228b3359459292c6e388a4f27222"
I20251026 09:09:06.233424 3949 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c232f8206f341968d7df85729f7eb38" candidate_uuid: "99a9020a509a4b69baa2c959c4e37450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14039bc98984406c9abec3ee17ad2879"
I20251026 09:09:06.233474 3799 raft_consensus.cc:3060] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.233472 3949 raft_consensus.cc:3060] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:06.234028 3949 raft_consensus.cc:2468] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 1.
I20251026 09:09:06.234051 3799 raft_consensus.cc:2468] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a9020a509a4b69baa2c959c4e37450 in term 1.
I20251026 09:09:06.234213 3839 leader_election.cc:304] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, 99a9020a509a4b69baa2c959c4e37450; no voters:
I20251026 09:09:06.234298 4003 raft_consensus.cc:2804] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:06.234361 4003 raft_consensus.cc:697] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 1 LEADER]: Becoming Leader. State: Replica: 99a9020a509a4b69baa2c959c4e37450, State: Running, Role: LEADER
I20251026 09:09:06.234414 4003 consensus_queue.cc:237] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:06.234987 3695 catalog_manager.cc:5649] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "99a9020a509a4b69baa2c959c4e37450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:06.284107 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:06.285658 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:06.285693 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:06.285820 4046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:06.286187 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:06.286309 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:06.286338 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:06.286351 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469746286352 us; error 0 us; skew 500 ppm
I20251026 09:09:06.287014 3128 webserver.cc:492] Webserver started at http://127.3.14.4:34935/ using document root <none> and password file <none>
I20251026 09:09:06.287083 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:06.287114 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:06.287168 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:06.287461 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-3-root/instance:
uuid: "af8f1aa87b8041adba8048ebc558fe32"
format_stamp: "Formatted at 2025-10-26 09:09:06 on dist-test-slave-w8k0"
I20251026 09:09:06.288350 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:06.288818 4051 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:06.288942 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:06.288988 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-3-root
uuid: "af8f1aa87b8041adba8048ebc558fe32"
format_stamp: "Formatted at 2025-10-26 09:09:06 on dist-test-slave-w8k0"
I20251026 09:09:06.289027 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761469742341867-3128-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:06.303495 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:06.303735 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:06.303879 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:06.304071 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:06.304102 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:06.304122 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:06.304136 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:06.307873 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.4:33287
I20251026 09:09:06.307935 4114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.4:33287 every 8 connection(s)
I20251026 09:09:06.310947 4115 heartbeater.cc:344] Connected to a master server at 127.3.14.62:43859
I20251026 09:09:06.311005 4115 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:06.311100 4115 heartbeater.cc:507] Master 127.3.14.62:43859 requested a full tablet report, sending...
I20251026 09:09:06.311379 3695 ts_manager.cc:194] Registered new tserver with Master: af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4:33287)
I20251026 09:09:06.311738 3695 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60068
I20251026 09:09:06.353462 4029 consensus_queue.cc:1048] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.354867 4120 consensus_queue.cc:1048] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.399541 4120 consensus_queue.cc:1048] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.401072 4011 consensus_queue.cc:1048] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.498740 4011 consensus_queue.cc:1048] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.500294 3999 consensus_queue.cc:1048] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.500547 4011 consensus_queue.cc:1048] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.501775 3999 consensus_queue.cc:1048] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.521854 4120 consensus_queue.cc:1048] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.523342 4029 consensus_queue.cc:1048] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.561182 4003 consensus_queue.cc:1048] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.562574 4008 consensus_queue.cc:1048] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.594729 4008 consensus_queue.cc:1048] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.596207 4003 consensus_queue.cc:1048] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.780314 4003 consensus_queue.cc:1048] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.781872 4008 consensus_queue.cc:1048] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:06.793893 3751 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:06.797053 3799 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "49c5577cdaab4885b395eab943d7ae8e"
dest_uuid: "c05e228b3359459292c6e388a4f27222"
mode: GRACEFUL
new_leader_uuid: "14039bc98984406c9abec3ee17ad2879"
from {username='slave'} at 127.0.0.1:50698
I20251026 09:09:06.797156 3799 raft_consensus.cc:606] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Received request to transfer leadership to TS 14039bc98984406c9abec3ee17ad2879
I20251026 09:09:06.797472 3751 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 09:09:06.797552 3751 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:07.095113 4120 raft_consensus.cc:993] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222: : Instructing follower 14039bc98984406c9abec3ee17ad2879 to start an election
I20251026 09:09:07.095214 4120 raft_consensus.cc:1081] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Signalling peer 14039bc98984406c9abec3ee17ad2879 to start an election
I20251026 09:09:07.095461 3949 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "49c5577cdaab4885b395eab943d7ae8e"
dest_uuid: "14039bc98984406c9abec3ee17ad2879"
from {username='slave'} at 127.0.0.1:58092
I20251026 09:09:07.095567 3949 raft_consensus.cc:493] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:09:07.095597 3949 raft_consensus.cc:3060] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:09:07.096196 3949 raft_consensus.cc:515] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:07.096338 3949 leader_election.cc:290] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [CANDIDATE]: Term 2 election: Requested vote from peers c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353), 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795)
I20251026 09:09:07.096480 3799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49c5577cdaab4885b395eab943d7ae8e" candidate_uuid: "14039bc98984406c9abec3ee17ad2879" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c05e228b3359459292c6e388a4f27222"
I20251026 09:09:07.096499 3874 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49c5577cdaab4885b395eab943d7ae8e" candidate_uuid: "14039bc98984406c9abec3ee17ad2879" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99a9020a509a4b69baa2c959c4e37450"
I20251026 09:09:07.096554 3799 raft_consensus.cc:3055] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:09:07.096565 3874 raft_consensus.cc:3060] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:09:07.096583 3799 raft_consensus.cc:740] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c05e228b3359459292c6e388a4f27222, State: Running, Role: LEADER
I20251026 09:09:07.096634 3799 consensus_queue.cc:260] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:07.096720 3799 raft_consensus.cc:3060] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:09:07.097306 3799 raft_consensus.cc:2468] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14039bc98984406c9abec3ee17ad2879 in term 2.
I20251026 09:09:07.097306 3874 raft_consensus.cc:2468] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14039bc98984406c9abec3ee17ad2879 in term 2.
I20251026 09:09:07.097512 3916 leader_election.cc:304] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14039bc98984406c9abec3ee17ad2879, 99a9020a509a4b69baa2c959c4e37450; no voters:
I20251026 09:09:07.097673 4129 raft_consensus.cc:2804] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:09:07.097735 4129 raft_consensus.cc:697] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 2 LEADER]: Becoming Leader. State: Replica: 14039bc98984406c9abec3ee17ad2879, State: Running, Role: LEADER
I20251026 09:09:07.097790 4129 consensus_queue.cc:237] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } }
I20251026 09:09:07.098273 3696 catalog_manager.cc:5649] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 reported cstate change: term changed from 1 to 2, leader changed from c05e228b3359459292c6e388a4f27222 (127.3.14.1) to 14039bc98984406c9abec3ee17ad2879 (127.3.14.3). New cstate: current_term: 2 leader_uuid: "14039bc98984406c9abec3ee17ad2879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: HEALTHY } } }
I20251026 09:09:07.312268 4115 heartbeater.cc:499] Master 127.3.14.62:43859 was elected leader, sending a full tablet report...
I20251026 09:09:07.608680 3874 raft_consensus.cc:1275] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 2 FOLLOWER]: Refusing update from remote peer 14039bc98984406c9abec3ee17ad2879: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:09:07.609018 4129 consensus_queue.cc:1048] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:07.610370 3799 raft_consensus.cc:1275] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 2 FOLLOWER]: Refusing update from remote peer 14039bc98984406c9abec3ee17ad2879: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:09:07.610602 4150 consensus_queue.cc:1048] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:08.797747 3751 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:08.797971 3751 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:08.798010 3751 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:10.798177 3751 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:10.798398 3751 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:10.798447 3751 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:11.802163 3949 consensus_queue.cc:237] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:11.802821 3874 raft_consensus.cc:1275] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Refusing update from remote peer 14039bc98984406c9abec3ee17ad2879: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:11.802821 3799 raft_consensus.cc:1275] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Refusing update from remote peer 14039bc98984406c9abec3ee17ad2879: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:11.803046 4159 consensus_queue.cc:1048] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:11.803169 4158 consensus_queue.cc:1048] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:11.803993 4150 raft_consensus.cc:2955] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.804119 3874 raft_consensus.cc:2955] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.804857 3799 raft_consensus.cc:2955] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.805984 3695 catalog_manager.cc:5649] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 reported cstate change: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "14039bc98984406c9abec3ee17ad2879" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 09:09:11.806454 3915 consensus_peers.cc:597] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 -> Peer af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4:33287): Couldn't send request to peer af8f1aa87b8041adba8048ebc558fe32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e8129b4c259448287c82d541517f061. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:09:11.809108 3874 consensus_queue.cc:237] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } attrs { replace: true } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:11.809675 3949 raft_consensus.cc:1275] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Refusing update from remote peer 99a9020a509a4b69baa2c959c4e37450: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:11.809717 3799 raft_consensus.cc:1275] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Refusing update from remote peer 99a9020a509a4b69baa2c959c4e37450: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:11.809881 4170 consensus_queue.cc:1048] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:11.809993 4171 consensus_queue.cc:1048] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:11.810685 4170 raft_consensus.cc:2955] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } attrs { replace: true } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.810822 3799 raft_consensus.cc:2955] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } attrs { replace: true } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.810848 3949 raft_consensus.cc:2955] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } attrs { replace: true } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.812364 3696 catalog_manager.cc:5649] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 reported cstate change: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "99a9020a509a4b69baa2c959c4e37450" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } attrs { replace: true } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
W20251026 09:09:11.812995 3840 consensus_peers.cc:597] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 -> Peer af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4:33287): Couldn't send request to peer af8f1aa87b8041adba8048ebc558fe32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50309fd6edc54037b485bcf80290eb89. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:09:11.815551 3799 consensus_queue.cc:237] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } attrs { replace: true } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:11.816103 3874 raft_consensus.cc:1275] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Refusing update from remote peer c05e228b3359459292c6e388a4f27222: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:11.816103 3949 raft_consensus.cc:1275] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Refusing update from remote peer c05e228b3359459292c6e388a4f27222: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:11.816326 4177 consensus_queue.cc:1048] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:11.816489 4178 consensus_queue.cc:1048] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:11.817175 4177 raft_consensus.cc:2955] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } attrs { replace: true } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.817337 3874 raft_consensus.cc:2955] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } attrs { replace: true } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.817369 3949 raft_consensus.cc:2955] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } attrs { replace: true } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.818389 3694 catalog_manager.cc:5649] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 reported cstate change: config changed from index -1 to 2, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "c05e228b3359459292c6e388a4f27222" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } attrs { replace: true } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.818954 3949 consensus_queue.cc:237] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:11.819499 3799 raft_consensus.cc:1275] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 2 FOLLOWER]: Refusing update from remote peer 14039bc98984406c9abec3ee17ad2879: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 09:09:11.819496 3874 raft_consensus.cc:1275] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 2 FOLLOWER]: Refusing update from remote peer 14039bc98984406c9abec3ee17ad2879: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20251026 09:09:11.819561 3764 consensus_peers.cc:597] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 -> Peer af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4:33287): Couldn't send request to peer af8f1aa87b8041adba8048ebc558fe32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e360a06882f64e0d92a723919a2cec55. This is attempt 1: this message will repeat every 5th retry.
W20251026 09:09:11.819808 3915 consensus_peers.cc:597] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 -> Peer af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4:33287): Couldn't send request to peer af8f1aa87b8041adba8048ebc558fe32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49c5577cdaab4885b395eab943d7ae8e. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:09:11.819808 4158 consensus_queue.cc:1048] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:09:11.819922 4159 consensus_queue.cc:1048] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:09:11.820544 4159 raft_consensus.cc:2955] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.820948 3874 raft_consensus.cc:2955] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:11.821691 3696 catalog_manager.cc:5649] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 reported cstate change: config changed from index -1 to 3, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New cstate: current_term: 2 leader_uuid: "14039bc98984406c9abec3ee17ad2879" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:11.821907 3799 raft_consensus.cc:2955] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER af8f1aa87b8041adba8048ebc558fe32 (127.3.14.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } } }
I20251026 09:09:12.221619 4185 ts_tablet_manager.cc:933] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: Initiating tablet copy from peer 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:12.221891 4185 tablet_copy_client.cc:323] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:42695
I20251026 09:09:12.224062 4188 ts_tablet_manager.cc:933] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: Initiating tablet copy from peer 99a9020a509a4b69baa2c959c4e37450 (127.3.14.2:34795)
I20251026 09:09:12.224249 4188 tablet_copy_client.cc:323] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:34795
I20251026 09:09:12.225162 3960 tablet_copy_service.cc:140] P 14039bc98984406c9abec3ee17ad2879: Received BeginTabletCopySession request for tablet 9e8129b4c259448287c82d541517f061 from peer af8f1aa87b8041adba8048ebc558fe32 ({username='slave'} at 127.0.0.1:43762)
I20251026 09:09:12.225232 3960 tablet_copy_service.cc:161] P 14039bc98984406c9abec3ee17ad2879: Beginning new tablet copy session on tablet 9e8129b4c259448287c82d541517f061 from peer af8f1aa87b8041adba8048ebc558fe32 at {username='slave'} at 127.0.0.1:43762: session id = af8f1aa87b8041adba8048ebc558fe32-9e8129b4c259448287c82d541517f061
I20251026 09:09:12.225971 3960 tablet_copy_source_session.cc:215] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:09:12.226466 4185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e8129b4c259448287c82d541517f061. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:12.228046 4185 tablet_copy_client.cc:806] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Starting download of 0 data blocks...
I20251026 09:09:12.228121 3885 tablet_copy_service.cc:140] P 99a9020a509a4b69baa2c959c4e37450: Received BeginTabletCopySession request for tablet 50309fd6edc54037b485bcf80290eb89 from peer af8f1aa87b8041adba8048ebc558fe32 ({username='slave'} at 127.0.0.1:44194)
I20251026 09:09:12.228174 3885 tablet_copy_service.cc:161] P 99a9020a509a4b69baa2c959c4e37450: Beginning new tablet copy session on tablet 50309fd6edc54037b485bcf80290eb89 from peer af8f1aa87b8041adba8048ebc558fe32 at {username='slave'} at 127.0.0.1:44194: session id = af8f1aa87b8041adba8048ebc558fe32-50309fd6edc54037b485bcf80290eb89
I20251026 09:09:12.228188 4185 tablet_copy_client.cc:670] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Starting download of 1 WAL segments...
I20251026 09:09:12.228657 3885 tablet_copy_source_session.cc:215] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:09:12.228942 4188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50309fd6edc54037b485bcf80290eb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:12.229211 4185 tablet_copy_client.cc:538] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:09:12.230229 4188 tablet_copy_client.cc:806] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Starting download of 0 data blocks...
I20251026 09:09:12.230340 4185 tablet_bootstrap.cc:492] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: Bootstrap starting.
I20251026 09:09:12.230388 4188 tablet_copy_client.cc:670] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Starting download of 1 WAL segments...
I20251026 09:09:12.230901 4188 tablet_copy_client.cc:538] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:09:12.231624 4188 tablet_bootstrap.cc:492] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: Bootstrap starting.
I20251026 09:09:12.232259 4185 tablet_bootstrap.cc:492] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:09:12.232513 4185 tablet_bootstrap.cc:492] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: Bootstrap complete.
I20251026 09:09:12.232717 4185 ts_tablet_manager.cc:1403] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.003s
I20251026 09:09:12.232919 4185 raft_consensus.cc:359] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:12.233021 4185 raft_consensus.cc:740] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: af8f1aa87b8041adba8048ebc558fe32, State: Initialized, Role: LEARNER
I20251026 09:09:12.233119 4188 tablet_bootstrap.cc:492] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:09:12.233091 4185 consensus_queue.cc:260] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:12.233285 4185 ts_tablet_manager.cc:1434] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.233323 4188 tablet_bootstrap.cc:492] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: Bootstrap complete.
I20251026 09:09:12.233404 4188 ts_tablet_manager.cc:1403] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:09:12.233552 4188 raft_consensus.cc:359] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } attrs { replace: true } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:12.233630 4188 raft_consensus.cc:740] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: af8f1aa87b8041adba8048ebc558fe32, State: Initialized, Role: LEARNER
I20251026 09:09:12.233685 4188 consensus_queue.cc:260] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } attrs { replace: true } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:12.233727 3960 tablet_copy_service.cc:342] P 14039bc98984406c9abec3ee17ad2879: Request end of tablet copy session af8f1aa87b8041adba8048ebc558fe32-9e8129b4c259448287c82d541517f061 received from {username='slave'} at 127.0.0.1:43762
I20251026 09:09:12.233909 3960 tablet_copy_service.cc:434] P 14039bc98984406c9abec3ee17ad2879: ending tablet copy session af8f1aa87b8041adba8048ebc558fe32-9e8129b4c259448287c82d541517f061 on tablet 9e8129b4c259448287c82d541517f061 with peer af8f1aa87b8041adba8048ebc558fe32
I20251026 09:09:12.233913 4188 ts_tablet_manager.cc:1434] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:09:12.234139 3885 tablet_copy_service.cc:342] P 99a9020a509a4b69baa2c959c4e37450: Request end of tablet copy session af8f1aa87b8041adba8048ebc558fe32-50309fd6edc54037b485bcf80290eb89 received from {username='slave'} at 127.0.0.1:44194
I20251026 09:09:12.234184 3885 tablet_copy_service.cc:434] P 99a9020a509a4b69baa2c959c4e37450: ending tablet copy session af8f1aa87b8041adba8048ebc558fe32-50309fd6edc54037b485bcf80290eb89 on tablet 50309fd6edc54037b485bcf80290eb89 with peer af8f1aa87b8041adba8048ebc558fe32
I20251026 09:09:12.281631 4188 ts_tablet_manager.cc:933] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: Initiating tablet copy from peer 14039bc98984406c9abec3ee17ad2879 (127.3.14.3:42695)
I20251026 09:09:12.281814 4188 tablet_copy_client.cc:323] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:42695
I20251026 09:09:12.282045 3960 tablet_copy_service.cc:140] P 14039bc98984406c9abec3ee17ad2879: Received BeginTabletCopySession request for tablet 49c5577cdaab4885b395eab943d7ae8e from peer af8f1aa87b8041adba8048ebc558fe32 ({username='slave'} at 127.0.0.1:43762)
I20251026 09:09:12.282126 3960 tablet_copy_service.cc:161] P 14039bc98984406c9abec3ee17ad2879: Beginning new tablet copy session on tablet 49c5577cdaab4885b395eab943d7ae8e from peer af8f1aa87b8041adba8048ebc558fe32 at {username='slave'} at 127.0.0.1:43762: session id = af8f1aa87b8041adba8048ebc558fe32-49c5577cdaab4885b395eab943d7ae8e
I20251026 09:09:12.282753 3960 tablet_copy_source_session.cc:215] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:09:12.283108 4188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c5577cdaab4885b395eab943d7ae8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:12.284564 4188 tablet_copy_client.cc:806] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Starting download of 0 data blocks...
I20251026 09:09:12.285401 4188 tablet_copy_client.cc:670] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Starting download of 1 WAL segments...
I20251026 09:09:12.285991 4188 tablet_copy_client.cc:538] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:09:12.286761 4188 tablet_bootstrap.cc:492] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: Bootstrap starting.
I20251026 09:09:12.288645 4188 tablet_bootstrap.cc:492] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:09:12.288857 4188 tablet_bootstrap.cc:492] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: Bootstrap complete.
I20251026 09:09:12.288961 4188 ts_tablet_manager.cc:1403] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.003s
I20251026 09:09:12.289147 4188 raft_consensus.cc:359] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:12.289238 4188 raft_consensus.cc:740] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: af8f1aa87b8041adba8048ebc558fe32, State: Initialized, Role: LEARNER
I20251026 09:09:12.289275 4188 consensus_queue.cc:260] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c05e228b3359459292c6e388a4f27222" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 35353 } } peers { permanent_uuid: "99a9020a509a4b69baa2c959c4e37450" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 34795 } attrs { replace: true } } peers { permanent_uuid: "14039bc98984406c9abec3ee17ad2879" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 42695 } } peers { permanent_uuid: "af8f1aa87b8041adba8048ebc558fe32" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33287 } attrs { promote: true } }
I20251026 09:09:12.289448 4188 ts_tablet_manager.cc:1434] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:09:12.289636 3960 tablet_copy_service.cc:342] P 14039bc98984406c9abec3ee17ad2879: Request end of tablet copy session af8f1aa87b8041adba8048ebc558fe32-49c5577cdaab4885b395eab943d7ae8e received from {username='slave'} at 127.0.0.1:43762
I20251026 09:09:12.289690 3960 tablet_copy_service.cc:434] P 14039bc98984406c9abec3ee17ad2879: ending tablet copy session af8f1aa87b8041adba8048ebc558fe32-49c5577cdaab4885b395eab943d7ae8e on tablet 49c5577cdaab4885b395eab943d7ae8e with peer af8f1aa87b8041adba8048ebc558fe32
I20251026 09:09:12.333021 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
W20251026 09:09:12.333572 3681 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20251026 09:09:12.334498 3744 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 09:09:12.334551 3744 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 09:09:12.336302 3914 consensus_peers.cc:597] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 -> Peer c05e228b3359459292c6e388a4f27222 (127.3.14.1:35353): Couldn't send request to peer c05e228b3359459292c6e388a4f27222. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:09:12.336313 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:12.336427 3128 tablet_replica.cc:333] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222: stopping tablet replica
I20251026 09:09:12.336484 3128 raft_consensus.cc:2243] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.336524 3128 raft_consensus.cc:2272] T dfc16b04f6174abdb51edf22f4abf664 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.336690 3128 tablet_replica.cc:333] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222: stopping tablet replica
I20251026 09:09:12.336728 3128 raft_consensus.cc:2243] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:12.336777 3128 raft_consensus.cc:2272] T 1f3bfdafccc4459ea2a3af896722f8d2 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.336928 3128 tablet_replica.cc:333] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222: stopping tablet replica
I20251026 09:09:12.336967 3128 raft_consensus.cc:2243] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.336999 3128 raft_consensus.cc:2272] T 49c5577cdaab4885b395eab943d7ae8e P c05e228b3359459292c6e388a4f27222 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.337199 3128 tablet_replica.cc:333] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222: stopping tablet replica
I20251026 09:09:12.337253 3128 raft_consensus.cc:2243] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:12.337333 3128 raft_consensus.cc:2272] T e360a06882f64e0d92a723919a2cec55 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.337518 3128 tablet_replica.cc:333] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222: stopping tablet replica
I20251026 09:09:12.337566 3128 raft_consensus.cc:2243] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:12.337638 3128 raft_consensus.cc:2272] T 62e4d046541948508a8dda9af5fbc6cb P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.337764 3128 tablet_replica.cc:333] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222: stopping tablet replica
I20251026 09:09:12.337805 3128 raft_consensus.cc:2243] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.337860 3128 raft_consensus.cc:2272] T 9e8129b4c259448287c82d541517f061 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.338101 3128 tablet_replica.cc:333] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222: stopping tablet replica
I20251026 09:09:12.338164 3128 raft_consensus.cc:2243] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.338212 3128 raft_consensus.cc:2272] T 0c232f8206f341968d7df85729f7eb38 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.338356 3128 tablet_replica.cc:333] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222: stopping tablet replica
I20251026 09:09:12.338400 3128 raft_consensus.cc:2243] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.338431 3128 raft_consensus.cc:2272] T 50309fd6edc54037b485bcf80290eb89 P c05e228b3359459292c6e388a4f27222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.350847 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:09:12.352151 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
W20251026 09:09:12.352882 3744 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 09:09:12.352974 3744 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 09:09:12.356356 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:12.356465 3128 tablet_replica.cc:333] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450: stopping tablet replica
I20251026 09:09:12.356529 3128 raft_consensus.cc:2243] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.356586 3128 raft_consensus.cc:2272] T e360a06882f64e0d92a723919a2cec55 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.356858 3128 tablet_replica.cc:333] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450: stopping tablet replica
I20251026 09:09:12.356910 3128 raft_consensus.cc:2243] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.356981 3128 raft_consensus.cc:2272] T 49c5577cdaab4885b395eab943d7ae8e P 99a9020a509a4b69baa2c959c4e37450 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.357177 3128 tablet_replica.cc:333] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450: stopping tablet replica
I20251026 09:09:12.357218 3128 raft_consensus.cc:2243] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:12.357304 3128 raft_consensus.cc:2272] T 50309fd6edc54037b485bcf80290eb89 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.357483 3128 tablet_replica.cc:333] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450: stopping tablet replica
I20251026 09:09:12.357534 3128 raft_consensus.cc:2243] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:12.357604 3128 raft_consensus.cc:2272] T 0c232f8206f341968d7df85729f7eb38 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.357722 3128 tablet_replica.cc:333] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450: stopping tablet replica
I20251026 09:09:12.357760 3128 raft_consensus.cc:2243] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.357821 3128 raft_consensus.cc:2272] T 62e4d046541948508a8dda9af5fbc6cb P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.357933 3128 tablet_replica.cc:333] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450: stopping tablet replica
I20251026 09:09:12.357969 3128 raft_consensus.cc:2243] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.358032 3128 raft_consensus.cc:2272] T 1f3bfdafccc4459ea2a3af896722f8d2 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.358170 3128 tablet_replica.cc:333] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450: stopping tablet replica
I20251026 09:09:12.358207 3128 raft_consensus.cc:2243] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:12.358301 3128 raft_consensus.cc:2272] T dfc16b04f6174abdb51edf22f4abf664 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.358412 3128 tablet_replica.cc:333] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450: stopping tablet replica
I20251026 09:09:12.358454 3128 raft_consensus.cc:2243] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.358528 3128 raft_consensus.cc:2272] T 9e8129b4c259448287c82d541517f061 P 99a9020a509a4b69baa2c959c4e37450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.370673 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:09:12.372439 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
W20251026 09:09:12.372936 3744 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 09:09:12.373049 3744 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 09:09:12.373452 3744 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 09:09:12.373505 3744 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 09:09:12.375929 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:12.376032 3128 tablet_replica.cc:333] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879: stopping tablet replica
I20251026 09:09:12.376099 3128 raft_consensus.cc:2243] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.376149 3128 raft_consensus.cc:2272] T dfc16b04f6174abdb51edf22f4abf664 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.376286 3128 tablet_replica.cc:333] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879: stopping tablet replica
I20251026 09:09:12.376318 3128 raft_consensus.cc:2243] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.376343 3128 raft_consensus.cc:2272] T e360a06882f64e0d92a723919a2cec55 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.376533 3128 tablet_replica.cc:333] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879: stopping tablet replica
I20251026 09:09:12.376572 3128 raft_consensus.cc:2243] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 2 LEADER]: Raft consensus shutting down.
I20251026 09:09:12.376632 3128 raft_consensus.cc:2272] T 49c5577cdaab4885b395eab943d7ae8e P 14039bc98984406c9abec3ee17ad2879 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.376786 3128 tablet_replica.cc:333] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879: stopping tablet replica
I20251026 09:09:12.376817 3128 raft_consensus.cc:2243] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.376840 3128 raft_consensus.cc:2272] T 0c232f8206f341968d7df85729f7eb38 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.376919 3128 tablet_replica.cc:333] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879: stopping tablet replica
I20251026 09:09:12.376945 3128 raft_consensus.cc:2243] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.376971 3128 raft_consensus.cc:2272] T 50309fd6edc54037b485bcf80290eb89 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.377146 3128 tablet_replica.cc:333] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879: stopping tablet replica
I20251026 09:09:12.377187 3128 raft_consensus.cc:2243] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.377207 3128 raft_consensus.cc:2272] T 62e4d046541948508a8dda9af5fbc6cb P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.377285 3128 tablet_replica.cc:333] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879: stopping tablet replica
I20251026 09:09:12.377310 3128 raft_consensus.cc:2243] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:12.377331 3128 raft_consensus.cc:2272] T 1f3bfdafccc4459ea2a3af896722f8d2 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.377416 3128 tablet_replica.cc:333] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879: stopping tablet replica
I20251026 09:09:12.377442 3128 raft_consensus.cc:2243] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:12.377488 3128 raft_consensus.cc:2272] T 9e8129b4c259448287c82d541517f061 P 14039bc98984406c9abec3ee17ad2879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.389564 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:09:12.390813 3128 tablet_server.cc:178] TabletServer@127.3.14.4:0 shutting down...
I20251026 09:09:12.393628 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:12.394050 3128 tablet_replica.cc:333] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32: stopping tablet replica
I20251026 09:09:12.394130 3128 raft_consensus.cc:2243] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32 [term 2 LEARNER]: Raft consensus shutting down.
I20251026 09:09:12.394184 3128 raft_consensus.cc:2272] T 49c5577cdaab4885b395eab943d7ae8e P af8f1aa87b8041adba8048ebc558fe32 [term 2 LEARNER]: Raft consensus is shut down!
I20251026 09:09:12.394420 3128 tablet_replica.cc:333] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32: stopping tablet replica
I20251026 09:09:12.394467 3128 raft_consensus.cc:2243] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32 [term 1 LEARNER]: Raft consensus shutting down.
I20251026 09:09:12.394513 3128 raft_consensus.cc:2272] T 9e8129b4c259448287c82d541517f061 P af8f1aa87b8041adba8048ebc558fe32 [term 1 LEARNER]: Raft consensus is shut down!
I20251026 09:09:12.394677 3128 tablet_replica.cc:333] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32: stopping tablet replica
I20251026 09:09:12.394714 3128 raft_consensus.cc:2243] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32 [term 1 LEARNER]: Raft consensus shutting down.
I20251026 09:09:12.394747 3128 raft_consensus.cc:2272] T 50309fd6edc54037b485bcf80290eb89 P af8f1aa87b8041adba8048ebc558fe32 [term 1 LEARNER]: Raft consensus is shut down!
I20251026 09:09:12.406566 3128 tablet_server.cc:195] TabletServer@127.3.14.4:0 shutdown complete.
I20251026 09:09:12.407634 3128 master.cc:561] Master@127.3.14.62:43859 shutting down...
I20251026 09:09:12.410283 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:12.410367 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:12.410396 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9b4bbe7796384fc6927d82ba13aca11c: stopping tablet replica
I20251026 09:09:12.422170 3128 master.cc:583] Master@127.3.14.62:43859 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7658 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20251026 09:09:12.430708 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:33485,127.3.14.61:46233,127.3.14.60:33801
I20251026 09:09:12.430888 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:12.432277 4199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.432437 4201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.432448 4198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:12.432543 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:12.432684 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:12.432708 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:12.432720 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469752432720 us; error 0 us; skew 500 ppm
I20251026 09:09:12.433342 3128 webserver.cc:492] Webserver started at http://127.3.14.62:35569/ using document root <none> and password file <none>
I20251026 09:09:12.433420 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:12.433472 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:12.433526 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:12.433794 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "24bfe89069694225840d97c9cdac80e7"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.434652 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:12.435070 4206 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.435220 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.435262 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "24bfe89069694225840d97c9cdac80e7"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.435353 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:12.450302 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:12.450562 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:12.453387 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:33485
I20251026 09:09:12.454227 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:09:12.458923 4258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:33485 every 8 connection(s)
I20251026 09:09:12.459112 4259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 09:09:12.459801 4261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.459896 4262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.459905 4264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:12.460016 4259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:12.461159 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:12.461275 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:12.461325 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:12.461356 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469752461356 us; error 0 us; skew 500 ppm
I20251026 09:09:12.461956 3128 webserver.cc:492] Webserver started at http://127.3.14.61:40873/ using document root <none> and password file <none>
I20251026 09:09:12.462045 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:12.462090 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:12.462155 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:12.462468 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root/instance:
uuid: "15c8f40392ee4969ab3a6f9b3c81893a"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.463366 4259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:12.463662 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
W20251026 09:09:12.463979 4259 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.14.61:46233: Network error: Client connection negotiation failed: client connection to 127.3.14.61:46233: connect: Connection refused (error 111)
I20251026 09:09:12.464170 4272 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.464284 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 09:09:12.464329 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root
uuid: "15c8f40392ee4969ab3a6f9b3c81893a"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.464377 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:12.482661 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:12.482887 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:12.485410 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.61:46233
I20251026 09:09:12.485448 4324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.61:46233 every 8 connection(s)
I20251026 09:09:12.485738 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:09:12.485738 4325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 09:09:12.486913 4327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:12.486864 4325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } has no permanent_uuid. Determining permanent_uuid...
W20251026 09:09:12.487118 4328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.487160 4330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:12.487501 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:12.487604 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:12.487632 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:12.487653 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469752487652 us; error 0 us; skew 500 ppm
I20251026 09:09:12.488234 3128 webserver.cc:492] Webserver started at http://127.3.14.60:45103/ using document root <none> and password file <none>
I20251026 09:09:12.488323 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:12.488381 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:12.488433 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:12.488693 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root/instance:
uuid: "f87af240b71b476ebccbc9378e97e8e0"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.489771 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 09:09:12.489953 4325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:12.490233 4336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.490366 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.490419 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root
uuid: "f87af240b71b476ebccbc9378e97e8e0"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.490470 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:12.493093 4325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } has no permanent_uuid. Determining permanent_uuid...
W20251026 09:09:12.493706 4325 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.14.60:33801: Network error: Client connection negotiation failed: client connection to 127.3.14.60:33801: connect: Connection refused (error 111)
I20251026 09:09:12.504076 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:12.504312 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:12.506865 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.60:33801
I20251026 09:09:12.506919 4390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.60:33801 every 8 connection(s)
I20251026 09:09:12.507076 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:09:12.507182 4391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:12.507866 4391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:12.510700 4391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:12.513341 4391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:12.516760 4391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0: Bootstrap starting.
I20251026 09:09:12.517058 4391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:12.517550 4391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0: No bootstrap required, opened a new log
I20251026 09:09:12.517673 4391 raft_consensus.cc:359] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:12.517730 4391 raft_consensus.cc:385] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:12.517745 4391 raft_consensus.cc:740] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87af240b71b476ebccbc9378e97e8e0, State: Initialized, Role: FOLLOWER
I20251026 09:09:12.517795 4391 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:12.517969 4391 sys_catalog.cc:565] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:12.517952 4396 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.518009 4396 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: This master's current role is: FOLLOWER
W20251026 09:09:12.519086 4407 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 09:09:12.519124 4407 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 09:09:12.529150 4259 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } attempt: 1
I20251026 09:09:12.532135 4259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } has no permanent_uuid. Determining permanent_uuid...
I20251026 09:09:12.535243 4259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7: Bootstrap starting.
I20251026 09:09:12.535773 4259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:12.536471 4259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7: No bootstrap required, opened a new log
I20251026 09:09:12.536654 4259 raft_consensus.cc:359] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:12.536720 4259 raft_consensus.cc:385] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:12.536747 4259 raft_consensus.cc:740] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24bfe89069694225840d97c9cdac80e7, State: Initialized, Role: FOLLOWER
I20251026 09:09:12.536799 4259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:12.536991 4259 sys_catalog.cc:565] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:12.536999 4409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.537097 4409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:12.538108 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20251026 09:09:12.538230 4420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 09:09:12.538270 4420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 09:09:12.549876 4325 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } attempt: 1
I20251026 09:09:12.553426 4325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a: Bootstrap starting.
I20251026 09:09:12.553867 4325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:12.554471 4325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a: No bootstrap required, opened a new log
I20251026 09:09:12.554636 4325 raft_consensus.cc:359] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:12.554701 4325 raft_consensus.cc:385] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:12.554728 4325 raft_consensus.cc:740] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c8f40392ee4969ab3a6f9b3c81893a, State: Initialized, Role: FOLLOWER
I20251026 09:09:12.554778 4325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:12.554955 4325 sys_catalog.cc:565] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:12.554970 4422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.555058 4422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:12.556066 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20251026 09:09:12.556145 4433 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 09:09:12.556200 4433 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 09:09:12.556321 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:12.557557 4434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:12.557668 3128 server_base.cc:1047] running on GCE node
W20251026 09:09:12.557832 4435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.557821 4437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:12.558029 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:12.558068 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:12.558087 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469752558087 us; error 0 us; skew 500 ppm
I20251026 09:09:12.558683 3128 webserver.cc:492] Webserver started at http://127.3.14.1:40169/ using document root <none> and password file <none>
I20251026 09:09:12.558766 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:12.558836 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:12.558888 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:12.559154 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "36cdf7e96f84494a99abe9f97a9bd8ee"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.560091 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:12.560545 4442 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.560690 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.560734 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "36cdf7e96f84494a99abe9f97a9bd8ee"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.560781 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:12.572885 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:12.573153 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:12.573328 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:12.573522 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:12.573550 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.573586 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:12.573621 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.577003 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:44027
I20251026 09:09:12.577037 4505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:44027 every 8 connection(s)
I20251026 09:09:12.577544 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:09:12.579305 3128 server_base.cc:1047] running on GCE node
W20251026 09:09:12.579577 4513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.580098 4514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.580269 4516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:12.580435 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:12.580482 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:12.580502 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469752580501 us; error 0 us; skew 500 ppm
I20251026 09:09:12.581044 4506 heartbeater.cc:344] Connected to a master server at 127.3.14.60:33801
I20251026 09:09:12.581111 4506 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:12.581184 3128 webserver.cc:492] Webserver started at http://127.3.14.2:40677/ using document root <none> and password file <none>
I20251026 09:09:12.581221 4506 heartbeater.cc:507] Master 127.3.14.60:33801 requested a full tablet report, sending...
I20251026 09:09:12.581265 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:12.581308 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:12.581377 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:12.581516 4355 ts_manager.cc:194] Registered new tserver with Master: 36cdf7e96f84494a99abe9f97a9bd8ee (127.3.14.1:44027)
I20251026 09:09:12.581697 4508 heartbeater.cc:344] Connected to a master server at 127.3.14.61:46233
I20251026 09:09:12.581766 4508 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:12.581707 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "f15a1a6a706944cea60fee169140109d"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.581875 4508 heartbeater.cc:507] Master 127.3.14.61:46233 requested a full tablet report, sending...
I20251026 09:09:12.581884 4507 heartbeater.cc:344] Connected to a master server at 127.3.14.62:33485
I20251026 09:09:12.581945 4507 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:12.582042 4507 heartbeater.cc:507] Master 127.3.14.62:33485 requested a full tablet report, sending...
I20251026 09:09:12.582134 4289 ts_manager.cc:194] Registered new tserver with Master: 36cdf7e96f84494a99abe9f97a9bd8ee (127.3.14.1:44027)
I20251026 09:09:12.582276 4223 ts_manager.cc:194] Registered new tserver with Master: 36cdf7e96f84494a99abe9f97a9bd8ee (127.3.14.1:44027)
I20251026 09:09:12.582829 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:12.583240 4521 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.583403 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.583456 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "f15a1a6a706944cea60fee169140109d"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.583482 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:12.594329 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:12.594563 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:12.594718 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:12.594923 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:12.594954 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.594987 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:12.595013 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.598242 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:39653
I20251026 09:09:12.598284 4584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:39653 every 8 connection(s)
I20251026 09:09:12.598724 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:12.600435 4593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.600922 4595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:12.601148 4592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:12.602378 4585 heartbeater.cc:344] Connected to a master server at 127.3.14.60:33801
I20251026 09:09:12.602459 4586 heartbeater.cc:344] Connected to a master server at 127.3.14.62:33485
I20251026 09:09:12.602535 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:12.602542 4586 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:12.602763 4586 heartbeater.cc:507] Master 127.3.14.62:33485 requested a full tablet report, sending...
I20251026 09:09:12.602782 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20251026 09:09:12.602819 4587 heartbeater.cc:344] Connected to a master server at 127.3.14.61:46233
I20251026 09:09:12.602465 4585 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:12.602880 4587 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:12.602954 4585 heartbeater.cc:507] Master 127.3.14.60:33801 requested a full tablet report, sending...
I20251026 09:09:12.602988 4587 heartbeater.cc:507] Master 127.3.14.61:46233 requested a full tablet report, sending...
I20251026 09:09:12.603032 4223 ts_manager.cc:194] Registered new tserver with Master: f15a1a6a706944cea60fee169140109d (127.3.14.2:39653)
W20251026 09:09:12.602821 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:12.603093 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469752603093 us; error 0 us; skew 500 ppm
I20251026 09:09:12.603188 4355 ts_manager.cc:194] Registered new tserver with Master: f15a1a6a706944cea60fee169140109d (127.3.14.2:39653)
I20251026 09:09:12.603200 4289 ts_manager.cc:194] Registered new tserver with Master: f15a1a6a706944cea60fee169140109d (127.3.14.2:39653)
I20251026 09:09:12.603711 3128 webserver.cc:492] Webserver started at http://127.3.14.3:39235/ using document root <none> and password file <none>
I20251026 09:09:12.603797 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:12.603852 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:12.603909 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:12.604162 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "4bddaf4cf5be42b09c0da48560edf9ea"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.605031 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:12.605470 4600 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.605595 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.605638 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "4bddaf4cf5be42b09c0da48560edf9ea"
format_stamp: "Formatted at 2025-10-26 09:09:12 on dist-test-slave-w8k0"
I20251026 09:09:12.605684 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:12.611987 4409 raft_consensus.cc:493] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:12.612059 4409 raft_consensus.cc:515] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:12.612202 4409 leader_election.cc:290] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15c8f40392ee4969ab3a6f9b3c81893a (127.3.14.61:46233), f87af240b71b476ebccbc9378e97e8e0 (127.3.14.60:33801)
I20251026 09:09:12.612412 4365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24bfe89069694225840d97c9cdac80e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87af240b71b476ebccbc9378e97e8e0" is_pre_election: true
I20251026 09:09:12.612409 4299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24bfe89069694225840d97c9cdac80e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" is_pre_election: true
I20251026 09:09:12.612500 4365 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24bfe89069694225840d97c9cdac80e7 in term 0.
I20251026 09:09:12.612521 4299 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24bfe89069694225840d97c9cdac80e7 in term 0.
I20251026 09:09:12.612668 4209 leader_election.cc:304] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c8f40392ee4969ab3a6f9b3c81893a, 24bfe89069694225840d97c9cdac80e7; no voters:
I20251026 09:09:12.612785 4409 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:12.612821 4409 raft_consensus.cc:493] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:12.612859 4409 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:12.613667 4409 raft_consensus.cc:515] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:12.613783 4409 leader_election.cc:290] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [CANDIDATE]: Term 1 election: Requested vote from peers 15c8f40392ee4969ab3a6f9b3c81893a (127.3.14.61:46233), f87af240b71b476ebccbc9378e97e8e0 (127.3.14.60:33801)
I20251026 09:09:12.613969 4299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24bfe89069694225840d97c9cdac80e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c8f40392ee4969ab3a6f9b3c81893a"
I20251026 09:09:12.613991 4365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24bfe89069694225840d97c9cdac80e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87af240b71b476ebccbc9378e97e8e0"
I20251026 09:09:12.614033 4299 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:12.614046 4365 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:12.614702 4299 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24bfe89069694225840d97c9cdac80e7 in term 1.
I20251026 09:09:12.614790 4365 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24bfe89069694225840d97c9cdac80e7 in term 1.
I20251026 09:09:12.614869 4209 leader_election.cc:304] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c8f40392ee4969ab3a6f9b3c81893a, 24bfe89069694225840d97c9cdac80e7; no voters:
I20251026 09:09:12.614952 4409 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:12.615087 4409 raft_consensus.cc:697] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 1 LEADER]: Becoming Leader. State: Replica: 24bfe89069694225840d97c9cdac80e7, State: Running, Role: LEADER
I20251026 09:09:12.615151 4409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:12.615484 4605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24bfe89069694225840d97c9cdac80e7. Latest consensus state: current_term: 1 leader_uuid: "24bfe89069694225840d97c9cdac80e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.615602 4605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:12.615762 4607 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:09:12.615870 4607 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:09:12.616382 4365 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 1 FOLLOWER]: Refusing update from remote peer 24bfe89069694225840d97c9cdac80e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:12.616431 4299 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 1 FOLLOWER]: Refusing update from remote peer 24bfe89069694225840d97c9cdac80e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:12.616567 4605 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:12.616647 4409 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:12.617141 4422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24bfe89069694225840d97c9cdac80e7. Latest consensus state: current_term: 1 leader_uuid: "24bfe89069694225840d97c9cdac80e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.617220 4422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:12.617264 4396 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24bfe89069694225840d97c9cdac80e7. Latest consensus state: current_term: 1 leader_uuid: "24bfe89069694225840d97c9cdac80e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.617326 4396 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:12.617925 4409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24bfe89069694225840d97c9cdac80e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.617990 4409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:12.618095 4396 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24bfe89069694225840d97c9cdac80e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.618180 4396 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:12.618314 4409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24bfe89069694225840d97c9cdac80e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.618373 4409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:12.618558 4422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24bfe89069694225840d97c9cdac80e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:12.618602 4422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:12.618377 4607 catalog_manager.cc:1357] Generated new cluster ID: f72a3ad358054cee819a2ac89028a79b
I20251026 09:09:12.618688 4607 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:09:12.627753 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:12.628018 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:12.628187 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:12.628392 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:12.628422 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.628454 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:12.628474 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:12.629364 4607 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:09:12.630015 4607 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:09:12.632205 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:37057
I20251026 09:09:12.632272 4675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:37057 every 8 connection(s)
I20251026 09:09:12.636504 4678 heartbeater.cc:344] Connected to a master server at 127.3.14.61:46233
I20251026 09:09:12.636572 4676 heartbeater.cc:344] Connected to a master server at 127.3.14.60:33801
I20251026 09:09:12.636582 4678 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:12.636646 4676 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:12.636740 4678 heartbeater.cc:507] Master 127.3.14.61:46233 requested a full tablet report, sending...
I20251026 09:09:12.636745 4676 heartbeater.cc:507] Master 127.3.14.60:33801 requested a full tablet report, sending...
I20251026 09:09:12.636598 4677 heartbeater.cc:344] Connected to a master server at 127.3.14.62:33485
I20251026 09:09:12.636860 4677 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:12.636945 4677 heartbeater.cc:507] Master 127.3.14.62:33485 requested a full tablet report, sending...
I20251026 09:09:12.637188 4289 ts_manager.cc:194] Registered new tserver with Master: 4bddaf4cf5be42b09c0da48560edf9ea (127.3.14.3:37057)
I20251026 09:09:12.637189 4355 ts_manager.cc:194] Registered new tserver with Master: 4bddaf4cf5be42b09c0da48560edf9ea (127.3.14.3:37057)
I20251026 09:09:12.637291 4223 ts_manager.cc:194] Registered new tserver with Master: 4bddaf4cf5be42b09c0da48560edf9ea (127.3.14.3:37057)
I20251026 09:09:12.637312 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004816419s
I20251026 09:09:12.649875 4607 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7: Generated new TSK 0
I20251026 09:09:12.650007 4607 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:09:13.519402 4407 catalog_manager.cc:1269] Loaded cluster ID: f72a3ad358054cee819a2ac89028a79b
I20251026 09:09:13.519480 4407 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0: loading cluster ID for follower catalog manager: success
I20251026 09:09:13.520202 4407 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0: acquiring CA information for follower catalog manager: success
I20251026 09:09:13.520311 4407 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 09:09:13.556420 4433 catalog_manager.cc:1269] Loaded cluster ID: f72a3ad358054cee819a2ac89028a79b
I20251026 09:09:13.556494 4433 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a: loading cluster ID for follower catalog manager: success
I20251026 09:09:13.557179 4433 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a: acquiring CA information for follower catalog manager: success
I20251026 09:09:13.557282 4433 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 09:09:13.583389 4223 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53338
I20251026 09:09:13.604173 4223 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53350
I20251026 09:09:13.638351 4223 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53362
I20251026 09:09:14.538138 4419 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:14.583927 4507 heartbeater.cc:499] Master 127.3.14.62:33485 was elected leader, sending a full tablet report...
I20251026 09:09:14.604729 4586 heartbeater.cc:499] Master 127.3.14.62:33485 was elected leader, sending a full tablet report...
I20251026 09:09:14.638940 4677 heartbeater.cc:499] Master 127.3.14.62:33485 was elected leader, sending a full tablet report...
I20251026 09:09:14.787653 3128 test_util.cc:276] Using random seed: 587482944
I20251026 09:09:14.793900 4223 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53376:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 09:09:14.800462 4640 tablet_service.cc:1505] Processing CreateTablet for tablet 2d0e07a977a1479789e504f7b199985b (DEFAULT_TABLE table=test-workload [id=036b82c6f1714a7ab0f082c702b8e019]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251026 09:09:14.800462 4470 tablet_service.cc:1505] Processing CreateTablet for tablet 2a4e73d327cb45a897d4b8054c15abf8 (DEFAULT_TABLE table=test-workload [id=036b82c6f1714a7ab0f082c702b8e019]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251026 09:09:14.800650 4640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d0e07a977a1479789e504f7b199985b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:14.800670 4470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4e73d327cb45a897d4b8054c15abf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:14.800726 4549 tablet_service.cc:1505] Processing CreateTablet for tablet 1813adc733684d758d289c37c6a81575 (DEFAULT_TABLE table=test-workload [id=036b82c6f1714a7ab0f082c702b8e019]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251026 09:09:14.800807 4549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1813adc733684d758d289c37c6a81575. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:14.801824 4713 tablet_bootstrap.cc:492] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee: Bootstrap starting.
I20251026 09:09:14.802256 4715 tablet_bootstrap.cc:492] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea: Bootstrap starting.
I20251026 09:09:14.802275 4713 tablet_bootstrap.cc:654] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:14.802686 4715 tablet_bootstrap.cc:654] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:14.803010 4713 tablet_bootstrap.cc:492] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee: No bootstrap required, opened a new log
I20251026 09:09:14.803013 4714 tablet_bootstrap.cc:492] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d: Bootstrap starting.
I20251026 09:09:14.803059 4713 ts_tablet_manager.cc:1403] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:14.803251 4713 raft_consensus.cc:359] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36cdf7e96f84494a99abe9f97a9bd8ee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 44027 } }
I20251026 09:09:14.803321 4713 raft_consensus.cc:385] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:14.803365 4713 raft_consensus.cc:740] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36cdf7e96f84494a99abe9f97a9bd8ee, State: Initialized, Role: FOLLOWER
I20251026 09:09:14.803426 4715 tablet_bootstrap.cc:492] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea: No bootstrap required, opened a new log
I20251026 09:09:14.803485 4714 tablet_bootstrap.cc:654] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:14.803524 4715 ts_tablet_manager.cc:1403] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:14.803449 4713 consensus_queue.cc:260] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36cdf7e96f84494a99abe9f97a9bd8ee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 44027 } }
I20251026 09:09:14.803612 4713 raft_consensus.cc:399] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:14.803668 4713 raft_consensus.cc:493] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:14.803699 4713 raft_consensus.cc:3060] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:14.803741 4715 raft_consensus.cc:359] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bddaf4cf5be42b09c0da48560edf9ea" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 37057 } }
I20251026 09:09:14.803810 4715 raft_consensus.cc:385] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:14.803835 4715 raft_consensus.cc:740] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bddaf4cf5be42b09c0da48560edf9ea, State: Initialized, Role: FOLLOWER
I20251026 09:09:14.803877 4715 consensus_queue.cc:260] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bddaf4cf5be42b09c0da48560edf9ea" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 37057 } }
I20251026 09:09:14.803941 4715 raft_consensus.cc:399] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:14.803972 4715 raft_consensus.cc:493] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:14.804004 4715 raft_consensus.cc:3060] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:14.804134 4714 tablet_bootstrap.cc:492] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d: No bootstrap required, opened a new log
I20251026 09:09:14.804178 4714 ts_tablet_manager.cc:1403] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:14.804311 4714 raft_consensus.cc:359] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15a1a6a706944cea60fee169140109d" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39653 } }
I20251026 09:09:14.804363 4714 raft_consensus.cc:385] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:14.804392 4714 raft_consensus.cc:740] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15a1a6a706944cea60fee169140109d, State: Initialized, Role: FOLLOWER
I20251026 09:09:14.804423 4713 raft_consensus.cc:515] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36cdf7e96f84494a99abe9f97a9bd8ee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 44027 } }
I20251026 09:09:14.804509 4714 consensus_queue.cc:260] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15a1a6a706944cea60fee169140109d" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39653 } }
I20251026 09:09:14.804522 4713 leader_election.cc:304] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36cdf7e96f84494a99abe9f97a9bd8ee; no voters:
I20251026 09:09:14.804577 4714 raft_consensus.cc:399] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:14.804634 4714 raft_consensus.cc:493] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:14.804695 4713 leader_election.cc:290] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:14.804739 4719 raft_consensus.cc:2804] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:14.804705 4714 raft_consensus.cc:3060] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:14.804904 4719 raft_consensus.cc:697] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 1 LEADER]: Becoming Leader. State: Replica: 36cdf7e96f84494a99abe9f97a9bd8ee, State: Running, Role: LEADER
I20251026 09:09:14.804870 4713 ts_tablet_manager.cc:1434] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 09:09:14.804970 4719 consensus_queue.cc:237] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36cdf7e96f84494a99abe9f97a9bd8ee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 44027 } }
I20251026 09:09:14.804729 4715 raft_consensus.cc:515] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bddaf4cf5be42b09c0da48560edf9ea" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 37057 } }
I20251026 09:09:14.805153 4715 leader_election.cc:304] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bddaf4cf5be42b09c0da48560edf9ea; no voters:
I20251026 09:09:14.805294 4715 leader_election.cc:290] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:14.805330 4223 catalog_manager.cc:5649] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 36cdf7e96f84494a99abe9f97a9bd8ee (127.3.14.1). New cstate: current_term: 1 leader_uuid: "36cdf7e96f84494a99abe9f97a9bd8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36cdf7e96f84494a99abe9f97a9bd8ee" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 44027 } health_report { overall_health: HEALTHY } } }
I20251026 09:09:14.805450 4715 ts_tablet_manager.cc:1434] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:09:14.807412 4724 raft_consensus.cc:2804] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:14.807556 4724 raft_consensus.cc:697] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 1 LEADER]: Becoming Leader. State: Replica: 4bddaf4cf5be42b09c0da48560edf9ea, State: Running, Role: LEADER
I20251026 09:09:14.807593 4714 raft_consensus.cc:515] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15a1a6a706944cea60fee169140109d" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39653 } }
I20251026 09:09:14.807684 4714 leader_election.cc:304] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f15a1a6a706944cea60fee169140109d; no voters:
I20251026 09:09:14.807804 4714 leader_election.cc:290] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:14.807854 4727 raft_consensus.cc:2804] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:14.808027 4714 ts_tablet_manager.cc:1434] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 09:09:14.807624 4724 consensus_queue.cc:237] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bddaf4cf5be42b09c0da48560edf9ea" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 37057 } }
I20251026 09:09:14.808068 4727 raft_consensus.cc:697] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 1 LEADER]: Becoming Leader. State: Replica: f15a1a6a706944cea60fee169140109d, State: Running, Role: LEADER
I20251026 09:09:14.808132 4727 consensus_queue.cc:237] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15a1a6a706944cea60fee169140109d" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39653 } }
I20251026 09:09:14.808575 4223 catalog_manager.cc:5649] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bddaf4cf5be42b09c0da48560edf9ea (127.3.14.3). New cstate: current_term: 1 leader_uuid: "4bddaf4cf5be42b09c0da48560edf9ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bddaf4cf5be42b09c0da48560edf9ea" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 37057 } health_report { overall_health: HEALTHY } } }
I20251026 09:09:14.809036 4222 catalog_manager.cc:5649] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d reported cstate change: term changed from 0 to 1, leader changed from <none> to f15a1a6a706944cea60fee169140109d (127.3.14.2). New cstate: current_term: 1 leader_uuid: "f15a1a6a706944cea60fee169140109d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15a1a6a706944cea60fee169140109d" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39653 } health_report { overall_health: HEALTHY } } }
I20251026 09:09:14.812803 3128 master.cc:561] Master@127.3.14.62:33485 shutting down...
I20251026 09:09:14.814917 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:14.814983 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:14.815004 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 24bfe89069694225840d97c9cdac80e7: stopping tablet replica
I20251026 09:09:14.827438 3128 master.cc:583] Master@127.3.14.62:33485 shutdown complete.
W20251026 09:09:15.807753 4507 heartbeater.cc:646] Failed to heartbeat to 127.3.14.62:33485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.14.62:33485: connect: Connection refused (error 111)
I20251026 09:09:16.310952 4737 raft_consensus.cc:493] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 24bfe89069694225840d97c9cdac80e7)
I20251026 09:09:16.311054 4737 raft_consensus.cc:515] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:16.311215 4738 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 24bfe89069694225840d97c9cdac80e7)
I20251026 09:09:16.311352 4738 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:16.311463 4737 leader_election.cc:290] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485), 15c8f40392ee4969ab3a6f9b3c81893a (127.3.14.61:46233)
I20251026 09:09:16.311538 4738 leader_election.cc:290] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485), f87af240b71b476ebccbc9378e97e8e0 (127.3.14.60:33801)
I20251026 09:09:16.311575 4299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87af240b71b476ebccbc9378e97e8e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" is_pre_election: true
I20251026 09:09:16.311646 4299 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f87af240b71b476ebccbc9378e97e8e0 in term 1.
I20251026 09:09:16.311841 4340 leader_election.cc:304] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c8f40392ee4969ab3a6f9b3c81893a, f87af240b71b476ebccbc9378e97e8e0; no voters:
I20251026 09:09:16.311868 4365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "f87af240b71b476ebccbc9378e97e8e0" is_pre_election: true
I20251026 09:09:16.311996 4365 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15c8f40392ee4969ab3a6f9b3c81893a in term 1.
I20251026 09:09:16.312036 4737 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 09:09:16.312100 4737 raft_consensus.cc:493] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 24bfe89069694225840d97c9cdac80e7)
I20251026 09:09:16.312144 4275 leader_election.cc:304] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c8f40392ee4969ab3a6f9b3c81893a, f87af240b71b476ebccbc9378e97e8e0; no voters:
I20251026 09:09:16.312145 4737 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 1 FOLLOWER]: Advancing to term 2
W20251026 09:09:16.312188 4338 leader_election.cc:336] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485): Network error: Client connection negotiation failed: client connection to 127.3.14.62:33485: connect: Connection refused (error 111)
W20251026 09:09:16.312263 4273 leader_election.cc:336] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485): Network error: Client connection negotiation failed: client connection to 127.3.14.62:33485: connect: Connection refused (error 111)
I20251026 09:09:16.312297 4738 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 09:09:16.312332 4738 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 24bfe89069694225840d97c9cdac80e7)
I20251026 09:09:16.312381 4738 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:09:16.313133 4738 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:16.313133 4737 raft_consensus.cc:515] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:16.313372 4737 leader_election.cc:290] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 2 election: Requested vote from peers 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485), 15c8f40392ee4969ab3a6f9b3c81893a (127.3.14.61:46233)
I20251026 09:09:16.313421 4299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87af240b71b476ebccbc9378e97e8e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "15c8f40392ee4969ab3a6f9b3c81893a"
I20251026 09:09:16.313536 4299 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f87af240b71b476ebccbc9378e97e8e0 in current term 2: Already voted for candidate 15c8f40392ee4969ab3a6f9b3c81893a in this term.
I20251026 09:09:16.313503 4365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "f87af240b71b476ebccbc9378e97e8e0"
I20251026 09:09:16.313469 4738 leader_election.cc:290] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [CANDIDATE]: Term 2 election: Requested vote from peers 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485), f87af240b71b476ebccbc9378e97e8e0 (127.3.14.60:33801)
I20251026 09:09:16.313601 4365 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 15c8f40392ee4969ab3a6f9b3c81893a in current term 2: Already voted for candidate f87af240b71b476ebccbc9378e97e8e0 in this term.
W20251026 09:09:16.313977 4338 leader_election.cc:336] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485): Network error: Client connection negotiation failed: client connection to 127.3.14.62:33485: connect: Connection refused (error 111)
W20251026 09:09:16.314018 4273 leader_election.cc:336] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485): Network error: Client connection negotiation failed: client connection to 127.3.14.62:33485: connect: Connection refused (error 111)
I20251026 09:09:16.314038 4338 leader_election.cc:304] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87af240b71b476ebccbc9378e97e8e0; no voters: 15c8f40392ee4969ab3a6f9b3c81893a, 24bfe89069694225840d97c9cdac80e7
I20251026 09:09:16.314049 4273 leader_election.cc:304] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15c8f40392ee4969ab3a6f9b3c81893a; no voters: 24bfe89069694225840d97c9cdac80e7, f87af240b71b476ebccbc9378e97e8e0
I20251026 09:09:16.314122 4737 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 09:09:16.314142 4738 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 09:09:18.034658 4749 raft_consensus.cc:493] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:18.034770 4749 raft_consensus.cc:515] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:18.034940 4749 leader_election.cc:290] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485), 15c8f40392ee4969ab3a6f9b3c81893a (127.3.14.61:46233)
I20251026 09:09:18.035211 4299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87af240b71b476ebccbc9378e97e8e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" is_pre_election: true
I20251026 09:09:18.035365 4299 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f87af240b71b476ebccbc9378e97e8e0 in term 2.
W20251026 09:09:18.035419 4338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.14.62:33485: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20251026 09:09:18.035548 4340 leader_election.cc:304] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c8f40392ee4969ab3a6f9b3c81893a, f87af240b71b476ebccbc9378e97e8e0; no voters:
I20251026 09:09:18.035660 4749 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 09:09:18.035713 4749 raft_consensus.cc:493] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:18.035741 4749 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 2 FOLLOWER]: Advancing to term 3
W20251026 09:09:18.035848 4338 leader_election.cc:336] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485): Network error: Client connection negotiation failed: client connection to 127.3.14.62:33485: connect: Connection refused (error 111)
I20251026 09:09:18.036592 4749 raft_consensus.cc:515] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:18.036748 4749 leader_election.cc:290] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 3 election: Requested vote from peers 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485), 15c8f40392ee4969ab3a6f9b3c81893a (127.3.14.61:46233)
I20251026 09:09:18.036931 4299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87af240b71b476ebccbc9378e97e8e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "15c8f40392ee4969ab3a6f9b3c81893a"
I20251026 09:09:18.037012 4299 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 2 FOLLOWER]: Advancing to term 3
W20251026 09:09:18.037263 4338 leader_election.cc:336] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485): Network error: Client connection negotiation failed: client connection to 127.3.14.62:33485: connect: Connection refused (error 111)
I20251026 09:09:18.037812 4299 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f87af240b71b476ebccbc9378e97e8e0 in term 3.
I20251026 09:09:18.038033 4340 leader_election.cc:304] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15c8f40392ee4969ab3a6f9b3c81893a, f87af240b71b476ebccbc9378e97e8e0; no voters: 24bfe89069694225840d97c9cdac80e7
I20251026 09:09:18.038129 4749 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:09:18.038285 4749 raft_consensus.cc:697] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 3 LEADER]: Becoming Leader. State: Replica: f87af240b71b476ebccbc9378e97e8e0, State: Running, Role: LEADER
I20251026 09:09:18.038371 4749 consensus_queue.cc:237] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } }
I20251026 09:09:18.038707 4752 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87af240b71b476ebccbc9378e97e8e0. Latest consensus state: current_term: 3 leader_uuid: "f87af240b71b476ebccbc9378e97e8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:18.038800 4752 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:18.038977 4754 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:09:18.039222 4754 catalog_manager.cc:679] Loaded metadata for table test-workload [id=036b82c6f1714a7ab0f082c702b8e019]
I20251026 09:09:18.039366 4754 tablet_loader.cc:96] loaded metadata for tablet 1813adc733684d758d289c37c6a81575 (table test-workload [id=036b82c6f1714a7ab0f082c702b8e019])
I20251026 09:09:18.039415 4754 tablet_loader.cc:96] loaded metadata for tablet 2a4e73d327cb45a897d4b8054c15abf8 (table test-workload [id=036b82c6f1714a7ab0f082c702b8e019])
I20251026 09:09:18.039446 4754 tablet_loader.cc:96] loaded metadata for tablet 2d0e07a977a1479789e504f7b199985b (table test-workload [id=036b82c6f1714a7ab0f082c702b8e019])
I20251026 09:09:18.039482 4754 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:09:18.039551 4754 catalog_manager.cc:1269] Loaded cluster ID: f72a3ad358054cee819a2ac89028a79b
I20251026 09:09:18.039577 4754 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:09:18.039757 4754 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:09:18.039830 4754 catalog_manager.cc:6033] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0: Loaded TSK: 0
I20251026 09:09:18.039944 4754 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251026 09:09:18.462280 4338 consensus_peers.cc:597] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 -> Peer 24bfe89069694225840d97c9cdac80e7 (127.3.14.62:33485): Couldn't send request to peer 24bfe89069694225840d97c9cdac80e7. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.62:33485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:09:18.519479 4406 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:18.519604 4406 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:18.532249 4299 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 3 FOLLOWER]: Refusing update from remote peer f87af240b71b476ebccbc9378e97e8e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20251026 09:09:18.532575 4752 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251026 09:09:18.533073 4755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87af240b71b476ebccbc9378e97e8e0. Latest consensus state: current_term: 3 leader_uuid: "f87af240b71b476ebccbc9378e97e8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:18.533159 4755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:18.533707 4749 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f87af240b71b476ebccbc9378e97e8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:18.533802 4749 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:18.534049 4755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f87af240b71b476ebccbc9378e97e8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24bfe89069694225840d97c9cdac80e7" member_type: VOTER last_known_addr { host: "127.3.14.62" port: 33485 } } peers { permanent_uuid: "15c8f40392ee4969ab3a6f9b3c81893a" member_type: VOTER last_known_addr { host: "127.3.14.61" port: 46233 } } peers { permanent_uuid: "f87af240b71b476ebccbc9378e97e8e0" member_type: VOTER last_known_addr { host: "127.3.14.60" port: 33801 } } }
I20251026 09:09:18.534124 4755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [sys.catalog]: This master's current role is: FOLLOWER
I20251026 09:09:18.650424 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:09:18.653270 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:18.653417 3128 tablet_replica.cc:333] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee: stopping tablet replica
I20251026 09:09:18.653481 3128 raft_consensus.cc:2243] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:18.653537 3128 raft_consensus.cc:2272] T 2a4e73d327cb45a897d4b8054c15abf8 P 36cdf7e96f84494a99abe9f97a9bd8ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:18.665620 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:09:18.666746 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:09:18.669847 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:18.670006 3128 tablet_replica.cc:333] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d: stopping tablet replica
I20251026 09:09:18.670081 3128 raft_consensus.cc:2243] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:18.670158 3128 raft_consensus.cc:2272] T 1813adc733684d758d289c37c6a81575 P f15a1a6a706944cea60fee169140109d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:18.682022 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:09:18.683087 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:09:18.685809 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:18.685945 3128 tablet_replica.cc:333] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea: stopping tablet replica
I20251026 09:09:18.686009 3128 raft_consensus.cc:2243] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:18.686065 3128 raft_consensus.cc:2272] T 2d0e07a977a1479789e504f7b199985b P 4bddaf4cf5be42b09c0da48560edf9ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:18.697898 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:09:18.698848 3128 master.cc:561] Master@127.3.14.61:46233 shutting down...
I20251026 09:09:18.701462 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:18.701557 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:18.701578 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 15c8f40392ee4969ab3a6f9b3c81893a: stopping tablet replica
I20251026 09:09:18.703302 3128 master.cc:583] Master@127.3.14.61:46233 shutdown complete.
I20251026 09:09:18.704401 3128 master.cc:561] Master@127.3.14.60:33801 shutting down...
I20251026 09:09:18.706851 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 3 LEADER]: Raft consensus shutting down.
I20251026 09:09:18.706952 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:18.707026 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P f87af240b71b476ebccbc9378e97e8e0: stopping tablet replica
I20251026 09:09:18.709056 3128 master.cc:583] Master@127.3.14.60:33801 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6283 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20251026 09:09:18.714359 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:36411
I20251026 09:09:18.714511 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:18.715893 4760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:18.715930 4762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:18.716053 4759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:18.716216 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:18.716317 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:18.716353 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:18.716373 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469758716374 us; error 0 us; skew 500 ppm
I20251026 09:09:18.716907 3128 webserver.cc:492] Webserver started at http://127.3.14.62:46315/ using document root <none> and password file <none>
I20251026 09:09:18.716974 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:18.717010 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:18.717047 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:18.717312 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3"
format_stamp: "Formatted at 2025-10-26 09:09:18 on dist-test-slave-w8k0"
I20251026 09:09:18.718143 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:18.718559 4767 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.718739 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.718793 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3"
format_stamp: "Formatted at 2025-10-26 09:09:18 on dist-test-slave-w8k0"
I20251026 09:09:18.718834 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:18.741832 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:18.742040 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:18.744596 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:36411
I20251026 09:09:18.747588 4819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:36411 every 8 connection(s)
I20251026 09:09:18.747768 4820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:18.748867 4820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3: Bootstrap starting.
I20251026 09:09:18.749168 4820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:18.749680 4820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3: No bootstrap required, opened a new log
I20251026 09:09:18.749809 4820 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3" member_type: VOTER }
I20251026 09:09:18.749862 4820 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:18.749886 4820 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec46dcda1e5a4635b34f36fd2dfb8ab3, State: Initialized, Role: FOLLOWER
I20251026 09:09:18.749944 4820 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3" member_type: VOTER }
I20251026 09:09:18.749979 4820 raft_consensus.cc:399] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:18.750001 4820 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:18.750034 4820 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:18.750530 4820 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3" member_type: VOTER }
I20251026 09:09:18.750590 4820 leader_election.cc:304] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec46dcda1e5a4635b34f36fd2dfb8ab3; no voters:
I20251026 09:09:18.750706 4820 leader_election.cc:290] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:18.750761 4823 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:18.750869 4823 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 1 LEADER]: Becoming Leader. State: Replica: ec46dcda1e5a4635b34f36fd2dfb8ab3, State: Running, Role: LEADER
I20251026 09:09:18.750876 4820 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:18.750977 4823 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3" member_type: VOTER }
I20251026 09:09:18.751205 4824 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3" member_type: VOTER } }
I20251026 09:09:18.751233 4825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec46dcda1e5a4635b34f36fd2dfb8ab3. Latest consensus state: current_term: 1 leader_uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec46dcda1e5a4635b34f36fd2dfb8ab3" member_type: VOTER } }
I20251026 09:09:18.751358 4824 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:18.751466 4825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:18.751663 4831 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:09:18.751811 4831 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:09:18.752403 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:09:18.752490 4831 catalog_manager.cc:1357] Generated new cluster ID: d621d659c160494bb6a50cd111f194d9
I20251026 09:09:18.752528 4831 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:09:18.778343 4831 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:09:18.778872 4831 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:09:18.782904 4831 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3: Generated new TSK 0
I20251026 09:09:18.783008 4831 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:09:18.784152 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:18.785851 4842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:18.785880 4841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:18.785943 4844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:18.786090 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:18.786180 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:18.786206 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:18.786218 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469758786219 us; error 0 us; skew 500 ppm
I20251026 09:09:18.786794 3128 webserver.cc:492] Webserver started at http://127.3.14.1:34999/ using document root <none> and password file <none>
I20251026 09:09:18.786880 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:18.786918 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:18.786959 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:18.787209 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "08195a01b17e4e4c97714b2ea8ac766e"
format_stamp: "Formatted at 2025-10-26 09:09:18 on dist-test-slave-w8k0"
I20251026 09:09:18.788137 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:18.788637 4849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.788754 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.788803 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "08195a01b17e4e4c97714b2ea8ac766e"
format_stamp: "Formatted at 2025-10-26 09:09:18 on dist-test-slave-w8k0"
I20251026 09:09:18.788854 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:18.797935 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:18.798173 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:18.798326 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:18.798535 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:18.798566 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.798600 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:18.798636 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.801460 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:41703
I20251026 09:09:18.801496 4912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:41703 every 8 connection(s)
I20251026 09:09:18.801848 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:18.803095 4917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:18.803124 4918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:18.803189 4920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:18.803514 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:18.803642 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:18.803673 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:18.803697 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469758803697 us; error 0 us; skew 500 ppm
I20251026 09:09:18.804349 3128 webserver.cc:492] Webserver started at http://127.3.14.2:35365/ using document root <none> and password file <none>
I20251026 09:09:18.804452 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:18.804504 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:18.804548 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:18.804816 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "dc99ba1b08c7464fa9046d22dab73fe5"
format_stamp: "Formatted at 2025-10-26 09:09:18 on dist-test-slave-w8k0"
I20251026 09:09:18.805953 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:18.806481 4925 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.806643 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.806686 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "dc99ba1b08c7464fa9046d22dab73fe5"
format_stamp: "Formatted at 2025-10-26 09:09:18 on dist-test-slave-w8k0"
I20251026 09:09:18.806742 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:18.809530 4913 heartbeater.cc:344] Connected to a master server at 127.3.14.62:36411
I20251026 09:09:18.809598 4913 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:18.809720 4913 heartbeater.cc:507] Master 127.3.14.62:36411 requested a full tablet report, sending...
I20251026 09:09:18.809945 4784 ts_manager.cc:194] Registered new tserver with Master: 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1:41703)
I20251026 09:09:18.810763 4784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41380
I20251026 09:09:18.817679 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:18.817922 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:18.818078 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:18.818275 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:18.818306 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.818341 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:18.818377 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.821164 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:42797
I20251026 09:09:18.821241 4988 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:42797 every 8 connection(s)
I20251026 09:09:18.821573 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:18.822889 4992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:18.823005 4995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:18.823511 4993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:18.823982 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:18.824189 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:18.824239 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:18.824260 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469758824260 us; error 0 us; skew 500 ppm
I20251026 09:09:18.824790 4989 heartbeater.cc:344] Connected to a master server at 127.3.14.62:36411
I20251026 09:09:18.824862 4989 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:18.824982 4989 heartbeater.cc:507] Master 127.3.14.62:36411 requested a full tablet report, sending...
I20251026 09:09:18.825045 3128 webserver.cc:492] Webserver started at http://127.3.14.3:37161/ using document root <none> and password file <none>
I20251026 09:09:18.825131 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:18.825183 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:18.825220 4784 ts_manager.cc:194] Registered new tserver with Master: dc99ba1b08c7464fa9046d22dab73fe5 (127.3.14.2:42797)
I20251026 09:09:18.825235 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:18.825573 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "4a4c5587cfb34855a8710a2b633cb762"
format_stamp: "Formatted at 2025-10-26 09:09:18 on dist-test-slave-w8k0"
I20251026 09:09:18.825599 4784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41382
I20251026 09:09:18.826576 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:18.827015 5000 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.827188 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.827242 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "4a4c5587cfb34855a8710a2b633cb762"
format_stamp: "Formatted at 2025-10-26 09:09:18 on dist-test-slave-w8k0"
I20251026 09:09:18.827301 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:18.839026 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:18.839295 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:18.839505 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:18.839728 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:18.839759 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.839798 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:18.839829 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:18.843036 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:44203
I20251026 09:09:18.843102 5063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:44203 every 8 connection(s)
I20251026 09:09:18.846380 5064 heartbeater.cc:344] Connected to a master server at 127.3.14.62:36411
I20251026 09:09:18.846455 5064 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:18.846572 5064 heartbeater.cc:507] Master 127.3.14.62:36411 requested a full tablet report, sending...
I20251026 09:09:18.846832 4784 ts_manager.cc:194] Registered new tserver with Master: 4a4c5587cfb34855a8710a2b633cb762 (127.3.14.3:44203)
I20251026 09:09:18.847196 4784 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41396
I20251026 09:09:18.847712 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004459269s
I20251026 09:09:19.811456 4913 heartbeater.cc:499] Master 127.3.14.62:36411 was elected leader, sending a full tablet report...
I20251026 09:09:19.826249 4989 heartbeater.cc:499] Master 127.3.14.62:36411 was elected leader, sending a full tablet report...
I20251026 09:09:19.847779 5064 heartbeater.cc:499] Master 127.3.14.62:36411 was elected leader, sending a full tablet report...
I20251026 09:09:19.873142 3128 test_util.cc:276] Using random seed: 592568434
I20251026 09:09:19.879122 4784 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41402:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:09:19.879307 4784 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:09:19.884788 5028 tablet_service.cc:1505] Processing CreateTablet for tablet 9ef1f88bf98f429f93adb6590312b4b9 (DEFAULT_TABLE table=test-workload [id=be1bfc6a9cee4a03b6bd51da1c0ccead]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20251026 09:09:19.884862 4953 tablet_service.cc:1505] Processing CreateTablet for tablet 9ef1f88bf98f429f93adb6590312b4b9 (DEFAULT_TABLE table=test-workload [id=be1bfc6a9cee4a03b6bd51da1c0ccead]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20251026 09:09:19.884994 5028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef1f88bf98f429f93adb6590312b4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:19.885015 4953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef1f88bf98f429f93adb6590312b4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:19.885313 4876 tablet_service.cc:1505] Processing CreateTablet for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c (DEFAULT_TABLE table=test-workload [id=be1bfc6a9cee4a03b6bd51da1c0ccead]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20251026 09:09:19.885318 4877 tablet_service.cc:1505] Processing CreateTablet for tablet 9ef1f88bf98f429f93adb6590312b4b9 (DEFAULT_TABLE table=test-workload [id=be1bfc6a9cee4a03b6bd51da1c0ccead]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20251026 09:09:19.885421 4876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:19.885546 4877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef1f88bf98f429f93adb6590312b4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:19.886804 4952 tablet_service.cc:1505] Processing CreateTablet for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c (DEFAULT_TABLE table=test-workload [id=be1bfc6a9cee4a03b6bd51da1c0ccead]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20251026 09:09:19.884788 5027 tablet_service.cc:1505] Processing CreateTablet for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c (DEFAULT_TABLE table=test-workload [id=be1bfc6a9cee4a03b6bd51da1c0ccead]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20251026 09:09:19.886900 4952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:19.886940 5027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:19.887850 5085 tablet_bootstrap.cc:492] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762: Bootstrap starting.
I20251026 09:09:19.888366 5085 tablet_bootstrap.cc:654] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:19.888690 5084 tablet_bootstrap.cc:492] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5: Bootstrap starting.
I20251026 09:09:19.888942 5085 tablet_bootstrap.cc:492] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762: No bootstrap required, opened a new log
I20251026 09:09:19.888988 5085 ts_tablet_manager.cc:1403] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:19.889087 5084 tablet_bootstrap.cc:654] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:19.889169 5085 raft_consensus.cc:359] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.889246 5085 raft_consensus.cc:385] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:19.889269 5085 raft_consensus.cc:740] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4c5587cfb34855a8710a2b633cb762, State: Initialized, Role: FOLLOWER
I20251026 09:09:19.889330 5085 consensus_queue.cc:260] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.889454 5086 tablet_bootstrap.cc:492] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e: Bootstrap starting.
I20251026 09:09:19.889565 5085 ts_tablet_manager.cc:1434] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:19.889654 5085 tablet_bootstrap.cc:492] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762: Bootstrap starting.
I20251026 09:09:19.889690 5084 tablet_bootstrap.cc:492] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5: No bootstrap required, opened a new log
I20251026 09:09:19.889791 5084 ts_tablet_manager.cc:1403] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:19.889961 5086 tablet_bootstrap.cc:654] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:19.889986 5084 raft_consensus.cc:359] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.890069 5084 raft_consensus.cc:385] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:19.890149 5084 raft_consensus.cc:740] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc99ba1b08c7464fa9046d22dab73fe5, State: Initialized, Role: FOLLOWER
I20251026 09:09:19.890273 5084 consensus_queue.cc:260] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.890498 5084 ts_tablet_manager.cc:1434] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:09:19.890604 5084 tablet_bootstrap.cc:492] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5: Bootstrap starting.
I20251026 09:09:19.890823 5086 tablet_bootstrap.cc:492] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e: No bootstrap required, opened a new log
I20251026 09:09:19.890925 5086 ts_tablet_manager.cc:1403] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:19.890094 5085 tablet_bootstrap.cc:654] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:19.891109 5086 raft_consensus.cc:359] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.891112 5084 tablet_bootstrap.cc:654] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:19.891244 5086 raft_consensus.cc:385] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:19.891360 5086 raft_consensus.cc:740] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08195a01b17e4e4c97714b2ea8ac766e, State: Initialized, Role: FOLLOWER
I20251026 09:09:19.891453 5086 consensus_queue.cc:260] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.891621 5086 ts_tablet_manager.cc:1434] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:19.891692 5086 tablet_bootstrap.cc:492] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e: Bootstrap starting.
I20251026 09:09:19.891935 5084 tablet_bootstrap.cc:492] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5: No bootstrap required, opened a new log
I20251026 09:09:19.892019 5084 ts_tablet_manager.cc:1403] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:19.892117 5086 tablet_bootstrap.cc:654] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:19.892179 5084 raft_consensus.cc:359] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.892251 5084 raft_consensus.cc:385] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:19.892274 5084 raft_consensus.cc:740] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc99ba1b08c7464fa9046d22dab73fe5, State: Initialized, Role: FOLLOWER
I20251026 09:09:19.892352 5084 consensus_queue.cc:260] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.892477 5084 ts_tablet_manager.cc:1434] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:19.892701 5086 tablet_bootstrap.cc:492] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e: No bootstrap required, opened a new log
I20251026 09:09:19.892750 5086 ts_tablet_manager.cc:1403] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:19.892787 5085 tablet_bootstrap.cc:492] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762: No bootstrap required, opened a new log
I20251026 09:09:19.892825 5085 ts_tablet_manager.cc:1403] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251026 09:09:19.892901 5086 raft_consensus.cc:359] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.892966 5086 raft_consensus.cc:385] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:19.892953 5085 raft_consensus.cc:359] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.892992 5086 raft_consensus.cc:740] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08195a01b17e4e4c97714b2ea8ac766e, State: Initialized, Role: FOLLOWER
I20251026 09:09:19.893005 5085 raft_consensus.cc:385] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:19.893025 5085 raft_consensus.cc:740] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4c5587cfb34855a8710a2b633cb762, State: Initialized, Role: FOLLOWER
I20251026 09:09:19.893038 5086 consensus_queue.cc:260] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.893070 5085 consensus_queue.cc:260] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:19.893142 5086 ts_tablet_manager.cc:1434] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:19.893160 5085 ts_tablet_manager.cc:1434] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:20.135037 5091 raft_consensus.cc:493] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:20.135174 5091 raft_consensus.cc:515] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:20.135416 5091 leader_election.cc:290] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1:41703), 4a4c5587cfb34855a8710a2b633cb762 (127.3.14.3:44203)
I20251026 09:09:20.138402 5038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ef1f88bf98f429f93adb6590312b4b9" candidate_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4c5587cfb34855a8710a2b633cb762" is_pre_election: true
I20251026 09:09:20.138415 4887 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ef1f88bf98f429f93adb6590312b4b9" candidate_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08195a01b17e4e4c97714b2ea8ac766e" is_pre_election: true
I20251026 09:09:20.138528 5038 raft_consensus.cc:2468] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc99ba1b08c7464fa9046d22dab73fe5 in term 0.
I20251026 09:09:20.138551 4887 raft_consensus.cc:2468] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc99ba1b08c7464fa9046d22dab73fe5 in term 0.
I20251026 09:09:20.138715 4926 leader_election.cc:304] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08195a01b17e4e4c97714b2ea8ac766e, dc99ba1b08c7464fa9046d22dab73fe5; no voters:
I20251026 09:09:20.138808 5091 raft_consensus.cc:2804] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:20.138872 5091 raft_consensus.cc:493] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:20.138893 5091 raft_consensus.cc:3060] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:20.139658 5091 raft_consensus.cc:515] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:20.139792 5091 leader_election.cc:290] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [CANDIDATE]: Term 1 election: Requested vote from peers 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1:41703), 4a4c5587cfb34855a8710a2b633cb762 (127.3.14.3:44203)
I20251026 09:09:20.140009 4887 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ef1f88bf98f429f93adb6590312b4b9" candidate_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08195a01b17e4e4c97714b2ea8ac766e"
I20251026 09:09:20.140009 5038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ef1f88bf98f429f93adb6590312b4b9" candidate_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4c5587cfb34855a8710a2b633cb762"
I20251026 09:09:20.140110 4887 raft_consensus.cc:3060] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:20.140121 5038 raft_consensus.cc:3060] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:20.140841 4887 raft_consensus.cc:2468] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc99ba1b08c7464fa9046d22dab73fe5 in term 1.
I20251026 09:09:20.140841 5038 raft_consensus.cc:2468] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc99ba1b08c7464fa9046d22dab73fe5 in term 1.
I20251026 09:09:20.141093 4928 leader_election.cc:304] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4c5587cfb34855a8710a2b633cb762, dc99ba1b08c7464fa9046d22dab73fe5; no voters:
I20251026 09:09:20.141206 5091 raft_consensus.cc:2804] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:20.141374 5091 raft_consensus.cc:697] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Becoming Leader. State: Replica: dc99ba1b08c7464fa9046d22dab73fe5, State: Running, Role: LEADER
I20251026 09:09:20.141440 5091 consensus_queue.cc:237] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:20.142021 4784 catalog_manager.cc:5649] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc99ba1b08c7464fa9046d22dab73fe5 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:20.154845 5091 raft_consensus.cc:493] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:09:20.154966 5091 raft_consensus.cc:515] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:20.155143 5091 leader_election.cc:290] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1:41703), 4a4c5587cfb34855a8710a2b633cb762 (127.3.14.3:44203)
I20251026 09:09:20.155465 4887 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23fd6ae8e1374e3bbde3ae682d73fb9c" candidate_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08195a01b17e4e4c97714b2ea8ac766e" is_pre_election: true
I20251026 09:09:20.155472 5038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23fd6ae8e1374e3bbde3ae682d73fb9c" candidate_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4c5587cfb34855a8710a2b633cb762" is_pre_election: true
I20251026 09:09:20.155557 4887 raft_consensus.cc:2468] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc99ba1b08c7464fa9046d22dab73fe5 in term 0.
I20251026 09:09:20.155611 5038 raft_consensus.cc:2468] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc99ba1b08c7464fa9046d22dab73fe5 in term 0.
I20251026 09:09:20.155743 4926 leader_election.cc:304] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08195a01b17e4e4c97714b2ea8ac766e, dc99ba1b08c7464fa9046d22dab73fe5; no voters:
I20251026 09:09:20.155857 5091 raft_consensus.cc:2804] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:09:20.155900 5091 raft_consensus.cc:493] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:09:20.155926 5091 raft_consensus.cc:3060] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:20.156608 5091 raft_consensus.cc:515] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:20.156750 5091 leader_election.cc:290] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [CANDIDATE]: Term 1 election: Requested vote from peers 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1:41703), 4a4c5587cfb34855a8710a2b633cb762 (127.3.14.3:44203)
I20251026 09:09:20.156960 4887 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23fd6ae8e1374e3bbde3ae682d73fb9c" candidate_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08195a01b17e4e4c97714b2ea8ac766e"
I20251026 09:09:20.156960 5038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23fd6ae8e1374e3bbde3ae682d73fb9c" candidate_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4c5587cfb34855a8710a2b633cb762"
I20251026 09:09:20.157074 4887 raft_consensus.cc:3060] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:20.157078 5038 raft_consensus.cc:3060] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:20.157913 4887 raft_consensus.cc:2468] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc99ba1b08c7464fa9046d22dab73fe5 in term 1.
I20251026 09:09:20.157913 5038 raft_consensus.cc:2468] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc99ba1b08c7464fa9046d22dab73fe5 in term 1.
I20251026 09:09:20.158164 4928 leader_election.cc:304] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4c5587cfb34855a8710a2b633cb762, dc99ba1b08c7464fa9046d22dab73fe5; no voters:
I20251026 09:09:20.158264 5091 raft_consensus.cc:2804] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:20.158330 5091 raft_consensus.cc:697] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Becoming Leader. State: Replica: dc99ba1b08c7464fa9046d22dab73fe5, State: Running, Role: LEADER
I20251026 09:09:20.158389 5091 consensus_queue.cc:237] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:20.159222 4784 catalog_manager.cc:5649] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc99ba1b08c7464fa9046d22dab73fe5 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:20.553767 5091 consensus_queue.cc:1048] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:20.555538 5104 consensus_queue.cc:1048] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:20.633172 5104 consensus_queue.cc:1048] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:20.635015 5091 consensus_queue.cc:1048] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:09:20.752493 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:20.756220 4963 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9ef1f88bf98f429f93adb6590312b4b9"
dest_uuid: "dc99ba1b08c7464fa9046d22dab73fe5"
mode: GRACEFUL
new_leader_uuid: "08195a01b17e4e4c97714b2ea8ac766e"
from {username='slave'} at 127.0.0.1:59746
I20251026 09:09:20.756369 4963 raft_consensus.cc:606] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Received request to transfer leadership to TS 08195a01b17e4e4c97714b2ea8ac766e
I20251026 09:09:20.756605 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 09:09:20.756672 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:21.091246 5091 raft_consensus.cc:993] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5: : Instructing follower 08195a01b17e4e4c97714b2ea8ac766e to start an election
I20251026 09:09:21.091358 5091 raft_consensus.cc:1081] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Signalling peer 08195a01b17e4e4c97714b2ea8ac766e to start an election
I20251026 09:09:21.091598 4887 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9ef1f88bf98f429f93adb6590312b4b9"
dest_uuid: "08195a01b17e4e4c97714b2ea8ac766e"
from {username='slave'} at 127.0.0.1:35614
I20251026 09:09:21.091686 4887 raft_consensus.cc:493] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:09:21.091709 4887 raft_consensus.cc:3060] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:09:21.092373 4887 raft_consensus.cc:515] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:21.092532 4887 leader_election.cc:290] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [CANDIDATE]: Term 2 election: Requested vote from peers dc99ba1b08c7464fa9046d22dab73fe5 (127.3.14.2:42797), 4a4c5587cfb34855a8710a2b633cb762 (127.3.14.3:44203)
I20251026 09:09:21.095381 5038 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ef1f88bf98f429f93adb6590312b4b9" candidate_uuid: "08195a01b17e4e4c97714b2ea8ac766e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a4c5587cfb34855a8710a2b633cb762"
I20251026 09:09:21.095381 4963 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ef1f88bf98f429f93adb6590312b4b9" candidate_uuid: "08195a01b17e4e4c97714b2ea8ac766e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dc99ba1b08c7464fa9046d22dab73fe5"
I20251026 09:09:21.095523 5038 raft_consensus.cc:3060] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:09:21.095527 4963 raft_consensus.cc:3055] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:09:21.095572 4963 raft_consensus.cc:740] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dc99ba1b08c7464fa9046d22dab73fe5, State: Running, Role: LEADER
I20251026 09:09:21.095654 4963 consensus_queue.cc:260] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:21.095749 4963 raft_consensus.cc:3060] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:09:21.096424 4963 raft_consensus.cc:2468] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08195a01b17e4e4c97714b2ea8ac766e in term 2.
I20251026 09:09:21.096424 5038 raft_consensus.cc:2468] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08195a01b17e4e4c97714b2ea8ac766e in term 2.
I20251026 09:09:21.096685 4852 leader_election.cc:304] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08195a01b17e4e4c97714b2ea8ac766e, 4a4c5587cfb34855a8710a2b633cb762; no voters:
I20251026 09:09:21.096838 5109 raft_consensus.cc:2804] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:09:21.096995 5109 raft_consensus.cc:697] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 2 LEADER]: Becoming Leader. State: Replica: 08195a01b17e4e4c97714b2ea8ac766e, State: Running, Role: LEADER
I20251026 09:09:21.097051 5109 consensus_queue.cc:237] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } }
I20251026 09:09:21.097572 4784 catalog_manager.cc:5649] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e reported cstate change: term changed from 1 to 2, leader changed from dc99ba1b08c7464fa9046d22dab73fe5 (127.3.14.2) to 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1). New cstate: current_term: 2 leader_uuid: "08195a01b17e4e4c97714b2ea8ac766e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } health_report { overall_health: UNKNOWN } } }
I20251026 09:09:21.480631 5038 raft_consensus.cc:1275] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 2 FOLLOWER]: Refusing update from remote peer 08195a01b17e4e4c97714b2ea8ac766e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:09:21.480932 5109 consensus_queue.cc:1048] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:21.482160 4963 raft_consensus.cc:1275] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 2 FOLLOWER]: Refusing update from remote peer 08195a01b17e4e4c97714b2ea8ac766e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:09:21.482458 5117 consensus_queue.cc:1048] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:22.756927 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:22.757115 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:22.757143 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:24.757336 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:24.757493 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:24.757521 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:26.757710 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:26.757861 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:26.757884 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:28.758035 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:28.758173 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:28.758193 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:30.758373 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:30.758524 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:30.758544 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:31.218680 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:31.220216 5132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:31.220381 5130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:31.220482 3128 server_base.cc:1047] running on GCE node
W20251026 09:09:31.220221 5129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:31.220702 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:31.220739 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:31.220753 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469771220754 us; error 0 us; skew 500 ppm
I20251026 09:09:31.221557 3128 webserver.cc:492] Webserver started at http://127.3.14.4:38775/ using document root <none> and password file <none>
I20251026 09:09:31.221647 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:31.221704 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:31.221752 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:31.222049 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-3-root/instance:
uuid: "a6bc212d070a4b848fc8ac9b65fce765"
format_stamp: "Formatted at 2025-10-26 09:09:31 on dist-test-slave-w8k0"
I20251026 09:09:31.222998 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:31.223610 5137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:31.223744 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:31.223795 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-3-root
uuid: "a6bc212d070a4b848fc8ac9b65fce765"
format_stamp: "Formatted at 2025-10-26 09:09:31 on dist-test-slave-w8k0"
I20251026 09:09:31.223848 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:31.241297 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:31.241531 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:31.241672 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:31.241886 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:31.241919 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:31.241945 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:31.241971 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:31.245496 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.4:43037
I20251026 09:09:31.245544 5200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.4:43037 every 8 connection(s)
I20251026 09:09:31.252813 5201 heartbeater.cc:344] Connected to a master server at 127.3.14.62:36411
I20251026 09:09:31.252936 5201 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:31.253072 5201 heartbeater.cc:507] Master 127.3.14.62:36411 requested a full tablet report, sending...
I20251026 09:09:31.253345 4783 ts_manager.cc:194] Registered new tserver with Master: a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4:43037)
I20251026 09:09:31.253774 4783 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60528
I20251026 09:09:31.759193 4963 consensus_queue.cc:237] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: true } }
I20251026 09:09:31.759979 4887 raft_consensus.cc:1275] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Refusing update from remote peer dc99ba1b08c7464fa9046d22dab73fe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:31.760016 5038 raft_consensus.cc:1275] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Refusing update from remote peer dc99ba1b08c7464fa9046d22dab73fe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:09:31.760281 5209 consensus_queue.cc:1048] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:31.760437 5208 consensus_queue.cc:1048] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:09:31.761286 5121 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: true } } }
I20251026 09:09:31.761384 5038 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: true } } }
I20251026 09:09:31.762573 4887 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: true } } }
I20251026 09:09:31.762789 4784 catalog_manager.cc:5649] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 reported cstate change: config changed from index -1 to 2, NON_VOTER a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: true } } }
W20251026 09:09:31.763585 4929 consensus_peers.cc:597] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 -> Peer a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4:43037): Couldn't send request to peer a6bc212d070a4b848fc8ac9b65fce765. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23fd6ae8e1374e3bbde3ae682d73fb9c. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:09:32.241811 5219 ts_tablet_manager.cc:933] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: Initiating tablet copy from peer dc99ba1b08c7464fa9046d22dab73fe5 (127.3.14.2:42797)
I20251026 09:09:32.242080 5219 tablet_copy_client.cc:323] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42797
I20251026 09:09:32.245525 4973 tablet_copy_service.cc:140] P dc99ba1b08c7464fa9046d22dab73fe5: Received BeginTabletCopySession request for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c from peer a6bc212d070a4b848fc8ac9b65fce765 ({username='slave'} at 127.0.0.1:45492)
I20251026 09:09:32.245626 4973 tablet_copy_service.cc:161] P dc99ba1b08c7464fa9046d22dab73fe5: Beginning new tablet copy session on tablet 23fd6ae8e1374e3bbde3ae682d73fb9c from peer a6bc212d070a4b848fc8ac9b65fce765 at {username='slave'} at 127.0.0.1:45492: session id = a6bc212d070a4b848fc8ac9b65fce765-23fd6ae8e1374e3bbde3ae682d73fb9c
I20251026 09:09:32.246333 4973 tablet_copy_source_session.cc:215] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:09:32.246714 5219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:32.248351 5219 tablet_copy_client.cc:806] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: tablet copy: Starting download of 0 data blocks...
I20251026 09:09:32.248526 5219 tablet_copy_client.cc:670] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: tablet copy: Starting download of 1 WAL segments...
I20251026 09:09:32.249145 5219 tablet_copy_client.cc:538] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:09:32.249945 5219 tablet_bootstrap.cc:492] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: Bootstrap starting.
I20251026 09:09:32.251760 5219 tablet_bootstrap.cc:492] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:09:32.251991 5219 tablet_bootstrap.cc:492] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: Bootstrap complete.
I20251026 09:09:32.252132 5219 ts_tablet_manager.cc:1403] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251026 09:09:32.252358 5219 raft_consensus.cc:359] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: true } }
I20251026 09:09:32.252434 5219 raft_consensus.cc:740] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a6bc212d070a4b848fc8ac9b65fce765, State: Initialized, Role: LEARNER
I20251026 09:09:32.252471 5219 consensus_queue.cc:260] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: true } }
I20251026 09:09:32.252683 5219 ts_tablet_manager.cc:1434] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:32.252719 5201 heartbeater.cc:499] Master 127.3.14.62:36411 was elected leader, sending a full tablet report...
I20251026 09:09:32.252913 4973 tablet_copy_service.cc:342] P dc99ba1b08c7464fa9046d22dab73fe5: Request end of tablet copy session a6bc212d070a4b848fc8ac9b65fce765-23fd6ae8e1374e3bbde3ae682d73fb9c received from {username='slave'} at 127.0.0.1:45492
I20251026 09:09:32.252970 4973 tablet_copy_service.cc:434] P dc99ba1b08c7464fa9046d22dab73fe5: ending tablet copy session a6bc212d070a4b848fc8ac9b65fce765-23fd6ae8e1374e3bbde3ae682d73fb9c on tablet 23fd6ae8e1374e3bbde3ae682d73fb9c with peer a6bc212d070a4b848fc8ac9b65fce765
I20251026 09:09:32.657492 5175 raft_consensus.cc:1217] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 09:09:32.758669 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:32.758812 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:32.758834 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:33.101835 5225 raft_consensus.cc:1064] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5: attempting to promote NON_VOTER a6bc212d070a4b848fc8ac9b65fce765 to VOTER
I20251026 09:09:33.102325 5225 consensus_queue.cc:237] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } }
I20251026 09:09:33.103080 4887 raft_consensus.cc:1275] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Refusing update from remote peer dc99ba1b08c7464fa9046d22dab73fe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:09:33.103044 5038 raft_consensus.cc:1275] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Refusing update from remote peer dc99ba1b08c7464fa9046d22dab73fe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:09:33.103034 5175 raft_consensus.cc:1275] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [term 1 LEARNER]: Refusing update from remote peer dc99ba1b08c7464fa9046d22dab73fe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:09:33.103308 5227 consensus_queue.cc:1048] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:09:33.103525 5227 consensus_queue.cc:1048] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:09:33.103662 5228 consensus_queue.cc:1048] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:09:33.104454 5225 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } } }
I20251026 09:09:33.104571 5175 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } } }
I20251026 09:09:33.104617 5038 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } } }
I20251026 09:09:33.105770 4887 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } } }
I20251026 09:09:33.105787 4783 catalog_manager.cc:5649] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 reported cstate change: config changed from index 2 to 3, a6bc212d070a4b848fc8ac9b65fce765 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "08195a01b17e4e4c97714b2ea8ac766e" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 41703 } attrs { replace: true } } peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } } }
I20251026 09:09:33.108299 4962 consensus_queue.cc:237] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } }
I20251026 09:09:33.108691 5175 raft_consensus.cc:1275] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [term 1 FOLLOWER]: Refusing update from remote peer dc99ba1b08c7464fa9046d22dab73fe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:09:33.108702 5038 raft_consensus.cc:1275] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Refusing update from remote peer dc99ba1b08c7464fa9046d22dab73fe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:09:33.108940 5227 consensus_queue.cc:1048] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:09:33.109051 5228 consensus_queue.cc:1048] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:09:33.109544 5227 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } } }
I20251026 09:09:33.109628 5175 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } } }
I20251026 09:09:33.110492 4770 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:09:33.110637 4782 catalog_manager.cc:5649] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 reported cstate change: config changed from index 3 to 4, VOTER 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1) evicted. New cstate: current_term: 1 leader_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } } }
I20251026 09:09:33.110729 5038 raft_consensus.cc:2955] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc99ba1b08c7464fa9046d22dab73fe5" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42797 } } peers { permanent_uuid: "4a4c5587cfb34855a8710a2b633cb762" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 44203 } } peers { permanent_uuid: "a6bc212d070a4b848fc8ac9b65fce765" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 43037 } attrs { promote: false } } }
I20251026 09:09:33.111740 4877 tablet_service.cc:1552] Processing DeleteTablet for tablet 23fd6ae8e1374e3bbde3ae682d73fb9c with delete_type TABLET_DATA_TOMBSTONED (TS 08195a01b17e4e4c97714b2ea8ac766e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35602
I20251026 09:09:33.112118 5237 tablet_replica.cc:333] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e: stopping tablet replica
I20251026 09:09:33.112183 5237 raft_consensus.cc:2243] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:33.112257 5237 raft_consensus.cc:2272] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:33.112504 5237 ts_tablet_manager.cc:1916] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:09:33.113752 5237 ts_tablet_manager.cc:1929] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 09:09:33.113811 5237 log.cc:1199] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 08195a01b17e4e4c97714b2ea8ac766e: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/23fd6ae8e1374e3bbde3ae682d73fb9c
I20251026 09:09:33.114210 4768 catalog_manager.cc:4985] TS 08195a01b17e4e4c97714b2ea8ac766e (127.3.14.1:41703): tablet 23fd6ae8e1374e3bbde3ae682d73fb9c (table test-workload [id=be1bfc6a9cee4a03b6bd51da1c0ccead]) successfully deleted
I20251026 09:09:34.759123 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:34.759502 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:34.759586 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:36.759812 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:36.760007 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:36.760051 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:38.760291 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:38.760476 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:38.760535 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:40.760766 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:40.760962 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:40.761049 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:42.761260 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:42.761440 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:42.761487 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:44.761690 4838 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:09:44.761863 4838 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:09:44.761902 4838 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:09:45.323048 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:09:45.326494 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:45.326650 3128 tablet_replica.cc:333] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e: stopping tablet replica
I20251026 09:09:45.326731 3128 raft_consensus.cc:2243] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 2 LEADER]: Raft consensus shutting down.
I20251026 09:09:45.326820 3128 raft_consensus.cc:2272] T 9ef1f88bf98f429f93adb6590312b4b9 P 08195a01b17e4e4c97714b2ea8ac766e [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:45.339198 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:09:45.340360 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:09:45.343569 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:45.343770 3128 tablet_replica.cc:333] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5: stopping tablet replica
I20251026 09:09:45.343825 3128 raft_consensus.cc:2243] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:45.343915 3128 raft_consensus.cc:2272] T 23fd6ae8e1374e3bbde3ae682d73fb9c P dc99ba1b08c7464fa9046d22dab73fe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:45.344103 3128 tablet_replica.cc:333] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5: stopping tablet replica
I20251026 09:09:45.344142 3128 raft_consensus.cc:2243] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:45.344187 3128 raft_consensus.cc:2272] T 9ef1f88bf98f429f93adb6590312b4b9 P dc99ba1b08c7464fa9046d22dab73fe5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:45.356192 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:09:45.357312 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:09:45.360930 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:45.361060 3128 tablet_replica.cc:333] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762: stopping tablet replica
I20251026 09:09:45.361132 3128 raft_consensus.cc:2243] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:45.361204 3128 raft_consensus.cc:2272] T 23fd6ae8e1374e3bbde3ae682d73fb9c P 4a4c5587cfb34855a8710a2b633cb762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:45.361375 3128 tablet_replica.cc:333] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762: stopping tablet replica
I20251026 09:09:45.361409 3128 raft_consensus.cc:2243] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:45.361433 3128 raft_consensus.cc:2272] T 9ef1f88bf98f429f93adb6590312b4b9 P 4a4c5587cfb34855a8710a2b633cb762 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:45.373253 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:09:45.374522 3128 tablet_server.cc:178] TabletServer@127.3.14.4:0 shutting down...
I20251026 09:09:45.377835 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:09:45.378008 3128 tablet_replica.cc:333] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765: stopping tablet replica
I20251026 09:09:45.378105 3128 raft_consensus.cc:2243] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:09:45.378177 3128 raft_consensus.cc:2272] T 23fd6ae8e1374e3bbde3ae682d73fb9c P a6bc212d070a4b848fc8ac9b65fce765 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:45.390028 3128 tablet_server.cc:195] TabletServer@127.3.14.4:0 shutdown complete.
I20251026 09:09:45.391064 3128 master.cc:561] Master@127.3.14.62:36411 shutting down...
I20251026 09:09:45.393905 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:45.394009 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:45.394042 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P ec46dcda1e5a4635b34f36fd2dfb8ab3: stopping tablet replica
I20251026 09:09:45.406567 3128 master.cc:583] Master@127.3.14.62:36411 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26697 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20251026 09:09:45.412250 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:37217
I20251026 09:09:45.412439 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:09:45.414080 3128 server_base.cc:1047] running on GCE node
W20251026 09:09:45.414148 5242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:45.414172 5245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:45.414186 5243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:45.414374 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:45.414415 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:45.414435 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469785414434 us; error 0 us; skew 500 ppm
I20251026 09:09:45.415091 3128 webserver.cc:492] Webserver started at http://127.3.14.62:38467/ using document root <none> and password file <none>
I20251026 09:09:45.415179 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:45.415227 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:45.415275 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:45.415609 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "6a9dbfa7222a45b48f1aca35ebc06bda"
format_stamp: "Formatted at 2025-10-26 09:09:45 on dist-test-slave-w8k0"
I20251026 09:09:45.416558 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:45.417018 5250 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:45.417146 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:45.417192 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "6a9dbfa7222a45b48f1aca35ebc06bda"
format_stamp: "Formatted at 2025-10-26 09:09:45 on dist-test-slave-w8k0"
I20251026 09:09:45.417248 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:45.428278 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:45.428531 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:45.431242 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:37217
I20251026 09:09:45.434015 5302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:37217 every 8 connection(s)
I20251026 09:09:45.434190 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:45.435637 5303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda: Bootstrap starting.
I20251026 09:09:45.435978 5303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:45.436525 5303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda: No bootstrap required, opened a new log
I20251026 09:09:45.436662 5303 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9dbfa7222a45b48f1aca35ebc06bda" member_type: VOTER }
I20251026 09:09:45.436733 5303 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:45.436748 5303 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9dbfa7222a45b48f1aca35ebc06bda, State: Initialized, Role: FOLLOWER
I20251026 09:09:45.436792 5303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9dbfa7222a45b48f1aca35ebc06bda" member_type: VOTER }
I20251026 09:09:45.436832 5303 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:45.436851 5303 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:45.436875 5303 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:45.437410 5303 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9dbfa7222a45b48f1aca35ebc06bda" member_type: VOTER }
I20251026 09:09:45.437474 5303 leader_election.cc:304] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a9dbfa7222a45b48f1aca35ebc06bda; no voters:
I20251026 09:09:45.437606 5303 leader_election.cc:290] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:45.437683 5306 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:45.437759 5303 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:45.437837 5306 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 1 LEADER]: Becoming Leader. State: Replica: 6a9dbfa7222a45b48f1aca35ebc06bda, State: Running, Role: LEADER
I20251026 09:09:45.437907 5306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9dbfa7222a45b48f1aca35ebc06bda" member_type: VOTER }
I20251026 09:09:45.438169 5308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a9dbfa7222a45b48f1aca35ebc06bda. Latest consensus state: current_term: 1 leader_uuid: "6a9dbfa7222a45b48f1aca35ebc06bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9dbfa7222a45b48f1aca35ebc06bda" member_type: VOTER } }
I20251026 09:09:45.438282 5308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:45.438176 5307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a9dbfa7222a45b48f1aca35ebc06bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9dbfa7222a45b48f1aca35ebc06bda" member_type: VOTER } }
I20251026 09:09:45.438575 5307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:45.438580 5312 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:09:45.438797 5312 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:09:45.439188 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:09:45.439401 5312 catalog_manager.cc:1357] Generated new cluster ID: 7675bf52633848c5ab1acb0d25748c74
I20251026 09:09:45.439453 5312 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:09:45.445701 5312 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:09:45.446130 5312 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:09:45.450819 5312 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda: Generated new TSK 0
I20251026 09:09:45.450948 5312 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:09:45.454686 3128 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 6.0601e-05s
I20251026 09:09:57.537366 3128 master.cc:561] Master@127.3.14.62:37217 shutting down...
I20251026 09:09:57.539994 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:09:57.540112 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:09:57.540146 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6a9dbfa7222a45b48f1aca35ebc06bda: stopping tablet replica
I20251026 09:09:57.551924 3128 master.cc:583] Master@127.3.14.62:37217 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12142 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20251026 09:09:57.554692 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:38039
I20251026 09:09:57.554875 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:57.556449 5328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:57.556527 5326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:57.556618 5325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:57.556629 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:57.556845 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:57.556892 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:57.556906 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469797556907 us; error 0 us; skew 500 ppm
I20251026 09:09:57.557590 3128 webserver.cc:492] Webserver started at http://127.3.14.62:36255/ using document root <none> and password file <none>
I20251026 09:09:57.557686 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:57.557737 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:57.557786 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:57.558099 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "cf188f15fe134b78bc311dad8ef2d599"
format_stamp: "Formatted at 2025-10-26 09:09:57 on dist-test-slave-w8k0"
I20251026 09:09:57.559026 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:57.559604 5333 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.559767 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.559804 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "cf188f15fe134b78bc311dad8ef2d599"
format_stamp: "Formatted at 2025-10-26 09:09:57 on dist-test-slave-w8k0"
I20251026 09:09:57.559864 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:57.583395 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:57.583638 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:57.586488 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:38039
I20251026 09:09:57.587122 5385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:38039 every 8 connection(s)
I20251026 09:09:57.587282 5386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:09:57.588354 5386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599: Bootstrap starting.
I20251026 09:09:57.588672 5386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599: Neither blocks nor log segments found. Creating new log.
I20251026 09:09:57.589192 5386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599: No bootstrap required, opened a new log
I20251026 09:09:57.589325 5386 raft_consensus.cc:359] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188f15fe134b78bc311dad8ef2d599" member_type: VOTER }
I20251026 09:09:57.589380 5386 raft_consensus.cc:385] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:09:57.589395 5386 raft_consensus.cc:740] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf188f15fe134b78bc311dad8ef2d599, State: Initialized, Role: FOLLOWER
I20251026 09:09:57.589457 5386 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188f15fe134b78bc311dad8ef2d599" member_type: VOTER }
I20251026 09:09:57.589497 5386 raft_consensus.cc:399] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:09:57.589514 5386 raft_consensus.cc:493] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:09:57.589532 5386 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:09:57.590056 5386 raft_consensus.cc:515] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188f15fe134b78bc311dad8ef2d599" member_type: VOTER }
I20251026 09:09:57.590118 5386 leader_election.cc:304] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf188f15fe134b78bc311dad8ef2d599; no voters:
I20251026 09:09:57.590230 5386 leader_election.cc:290] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:09:57.590296 5389 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:09:57.590389 5386 sys_catalog.cc:565] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:09:57.590451 5389 raft_consensus.cc:697] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 1 LEADER]: Becoming Leader. State: Replica: cf188f15fe134b78bc311dad8ef2d599, State: Running, Role: LEADER
I20251026 09:09:57.590505 5389 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188f15fe134b78bc311dad8ef2d599" member_type: VOTER }
I20251026 09:09:57.590730 5390 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf188f15fe134b78bc311dad8ef2d599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188f15fe134b78bc311dad8ef2d599" member_type: VOTER } }
I20251026 09:09:57.590797 5391 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf188f15fe134b78bc311dad8ef2d599. Latest consensus state: current_term: 1 leader_uuid: "cf188f15fe134b78bc311dad8ef2d599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188f15fe134b78bc311dad8ef2d599" member_type: VOTER } }
I20251026 09:09:57.590876 5391 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:57.590798 5390 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [sys.catalog]: This master's current role is: LEADER
I20251026 09:09:57.591259 5399 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:09:57.591424 5399 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:09:57.591758 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:09:57.592024 5399 catalog_manager.cc:1357] Generated new cluster ID: 64d424f7e6194306b7201f103fcf4c57
I20251026 09:09:57.592065 5399 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:09:57.601544 5399 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:09:57.601994 5399 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:09:57.609884 5399 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599: Generated new TSK 0
I20251026 09:09:57.609989 5399 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:09:57.623940 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:57.625423 5407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:57.625478 5410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:57.625523 3128 server_base.cc:1047] running on GCE node
W20251026 09:09:57.625682 5408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:57.625823 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:57.625865 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:57.625893 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469797625893 us; error 0 us; skew 500 ppm
I20251026 09:09:57.626516 3128 webserver.cc:492] Webserver started at http://127.3.14.1:35113/ using document root <none> and password file <none>
I20251026 09:09:57.626607 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:57.626693 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:57.626750 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:57.627075 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "0d6a898d8ba2431da2d5414893758168"
format_stamp: "Formatted at 2025-10-26 09:09:57 on dist-test-slave-w8k0"
I20251026 09:09:57.628126 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:57.628619 5415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.628758 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.628806 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "0d6a898d8ba2431da2d5414893758168"
format_stamp: "Formatted at 2025-10-26 09:09:57 on dist-test-slave-w8k0"
I20251026 09:09:57.628862 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:57.641403 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:57.641685 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:57.641858 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:57.642071 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:57.642102 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.642135 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:57.642163 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.645107 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:34493
I20251026 09:09:57.645146 5478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:34493 every 8 connection(s)
I20251026 09:09:57.645514 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:57.647022 5484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:57.647042 5486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:57.647152 5483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:57.647655 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:57.647802 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:57.647833 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:57.647850 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469797647850 us; error 0 us; skew 500 ppm
I20251026 09:09:57.648653 3128 webserver.cc:492] Webserver started at http://127.3.14.2:43839/ using document root <none> and password file <none>
I20251026 09:09:57.648753 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:57.648804 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:57.648856 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:57.649152 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "037de86768b44e0db081bbd97bf22b70"
format_stamp: "Formatted at 2025-10-26 09:09:57 on dist-test-slave-w8k0"
I20251026 09:09:57.649703 5479 heartbeater.cc:344] Connected to a master server at 127.3.14.62:38039
I20251026 09:09:57.649771 5479 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:57.649889 5479 heartbeater.cc:507] Master 127.3.14.62:38039 requested a full tablet report, sending...
I20251026 09:09:57.650161 5350 ts_manager.cc:194] Registered new tserver with Master: 0d6a898d8ba2431da2d5414893758168 (127.3.14.1:34493)
I20251026 09:09:57.650236 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:09:57.650739 5491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.650753 5350 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60864
I20251026 09:09:57.650882 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.650938 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "037de86768b44e0db081bbd97bf22b70"
format_stamp: "Formatted at 2025-10-26 09:09:57 on dist-test-slave-w8k0"
I20251026 09:09:57.650996 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:57.657992 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:57.658231 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:57.658370 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:57.658573 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:57.658605 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.658630 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:57.658666 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.661895 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:38981
I20251026 09:09:57.661944 5554 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:38981 every 8 connection(s)
I20251026 09:09:57.662340 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:09:57.663827 5558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:57.663849 5561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:09:57.663926 5559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:09:57.664170 3128 server_base.cc:1047] running on GCE node
I20251026 09:09:57.664355 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:09:57.664391 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:09:57.664404 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469797664405 us; error 0 us; skew 500 ppm
I20251026 09:09:57.665161 3128 webserver.cc:492] Webserver started at http://127.3.14.3:45151/ using document root <none> and password file <none>
I20251026 09:09:57.665271 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:09:57.665334 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:09:57.665395 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:09:57.665755 5555 heartbeater.cc:344] Connected to a master server at 127.3.14.62:38039
I20251026 09:09:57.665747 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "bb2825f9f74f4bd1af6d9e6ea98ad977"
format_stamp: "Formatted at 2025-10-26 09:09:57 on dist-test-slave-w8k0"
I20251026 09:09:57.665858 5555 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:57.665972 5555 heartbeater.cc:507] Master 127.3.14.62:38039 requested a full tablet report, sending...
I20251026 09:09:57.666215 5350 ts_manager.cc:194] Registered new tserver with Master: 037de86768b44e0db081bbd97bf22b70 (127.3.14.2:38981)
I20251026 09:09:57.666570 5350 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60866
I20251026 09:09:57.666877 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:09:57.667407 5566 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.667537 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.667584 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "bb2825f9f74f4bd1af6d9e6ea98ad977"
format_stamp: "Formatted at 2025-10-26 09:09:57 on dist-test-slave-w8k0"
I20251026 09:09:57.667640 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:09:57.675621 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:09:57.675925 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:09:57.676111 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:09:57.676329 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:09:57.676360 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.676398 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:09:57.676424 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:09:57.679873 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:41683
I20251026 09:09:57.679922 5629 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:41683 every 8 connection(s)
I20251026 09:09:57.683421 5630 heartbeater.cc:344] Connected to a master server at 127.3.14.62:38039
I20251026 09:09:57.683533 5630 heartbeater.cc:461] Registering TS with master...
I20251026 09:09:57.683674 5630 heartbeater.cc:507] Master 127.3.14.62:38039 requested a full tablet report, sending...
I20251026 09:09:57.683972 5350 ts_manager.cc:194] Registered new tserver with Master: bb2825f9f74f4bd1af6d9e6ea98ad977 (127.3.14.3:41683)
I20251026 09:09:57.684316 5350 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60872
I20251026 09:09:57.684587 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00451134s
I20251026 09:09:58.651463 5479 heartbeater.cc:499] Master 127.3.14.62:38039 was elected leader, sending a full tablet report...
I20251026 09:09:58.667171 5555 heartbeater.cc:499] Master 127.3.14.62:38039 was elected leader, sending a full tablet report...
I20251026 09:09:58.684983 5630 heartbeater.cc:499] Master 127.3.14.62:38039 was elected leader, sending a full tablet report...
I20251026 09:09:59.591861 5404 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:01.592121 5404 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:03.592391 5404 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:05.592660 5404 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:07.592890 5404 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:09.593182 5404 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:09.769024 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:10:09.772277 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:09.784132 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:10:09.785097 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:10:09.789395 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:09.800962 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:10:09.801911 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:10:09.805575 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:09.817416 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:10:09.818302 3128 master.cc:561] Master@127.3.14.62:38039 shutting down...
I20251026 09:10:09.821491 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:10:09.821588 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:09.821610 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P cf188f15fe134b78bc311dad8ef2d599: stopping tablet replica
I20251026 09:10:09.834242 3128 master.cc:583] Master@127.3.14.62:38039 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12283 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20251026 09:10:09.838224 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:34649
I20251026 09:10:09.838390 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:09.839731 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:09.839777 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:09.839803 5634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:09.839902 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:09.840119 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:09.840162 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:09.840176 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469809840177 us; error 0 us; skew 500 ppm
I20251026 09:10:09.840924 3128 webserver.cc:492] Webserver started at http://127.3.14.62:41517/ using document root <none> and password file <none>
I20251026 09:10:09.841024 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:09.841066 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:09.841125 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:09.841423 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "18a6444240cf41bf9311ef8ec754b494"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.842576 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:09.843304 5642 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.843465 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.843518 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "18a6444240cf41bf9311ef8ec754b494"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.843577 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:09.856523 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:09.856781 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:09.859531 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:34649
I20251026 09:10:09.861418 5694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:34649 every 8 connection(s)
I20251026 09:10:09.861570 5695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:09.862675 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494: Bootstrap starting.
I20251026 09:10:09.863005 5695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:09.863636 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494: No bootstrap required, opened a new log
I20251026 09:10:09.863793 5695 raft_consensus.cc:359] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a6444240cf41bf9311ef8ec754b494" member_type: VOTER }
I20251026 09:10:09.863847 5695 raft_consensus.cc:385] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:09.863860 5695 raft_consensus.cc:740] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a6444240cf41bf9311ef8ec754b494, State: Initialized, Role: FOLLOWER
I20251026 09:10:09.863904 5695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a6444240cf41bf9311ef8ec754b494" member_type: VOTER }
I20251026 09:10:09.863976 5695 raft_consensus.cc:399] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:10:09.864004 5695 raft_consensus.cc:493] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:10:09.864028 5695 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:09.864559 5695 raft_consensus.cc:515] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a6444240cf41bf9311ef8ec754b494" member_type: VOTER }
I20251026 09:10:09.864624 5695 leader_election.cc:304] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18a6444240cf41bf9311ef8ec754b494; no voters:
I20251026 09:10:09.864727 5695 leader_election.cc:290] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:10:09.864804 5698 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:09.864878 5695 sys_catalog.cc:565] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:10:09.864987 5698 raft_consensus.cc:697] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 1 LEADER]: Becoming Leader. State: Replica: 18a6444240cf41bf9311ef8ec754b494, State: Running, Role: LEADER
I20251026 09:10:09.865052 5698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a6444240cf41bf9311ef8ec754b494" member_type: VOTER }
I20251026 09:10:09.865317 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18a6444240cf41bf9311ef8ec754b494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a6444240cf41bf9311ef8ec754b494" member_type: VOTER } }
I20251026 09:10:09.865422 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [sys.catalog]: This master's current role is: LEADER
I20251026 09:10:09.865339 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18a6444240cf41bf9311ef8ec754b494. Latest consensus state: current_term: 1 leader_uuid: "18a6444240cf41bf9311ef8ec754b494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a6444240cf41bf9311ef8ec754b494" member_type: VOTER } }
I20251026 09:10:09.865512 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [sys.catalog]: This master's current role is: LEADER
I20251026 09:10:09.865958 5707 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:10:09.866155 5707 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:10:09.866287 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:10:09.866909 5707 catalog_manager.cc:1357] Generated new cluster ID: 41f987d10dae4df0af8a66f1d4eeeec1
I20251026 09:10:09.866950 5707 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:10:09.874823 5707 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:10:09.875262 5707 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:10:09.880533 5707 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494: Generated new TSK 0
I20251026 09:10:09.880640 5707 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:10:09.882004 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:09.883533 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:09.883533 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:09.883646 3128 server_base.cc:1047] running on GCE node
W20251026 09:10:09.883771 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:09.883879 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:09.883940 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:09.883966 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469809883966 us; error 0 us; skew 500 ppm
I20251026 09:10:09.884629 3128 webserver.cc:492] Webserver started at http://127.3.14.1:38469/ using document root <none> and password file <none>
I20251026 09:10:09.884722 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:09.884759 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:09.884799 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:09.885092 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "4ff4051b9feb40299a97d93f99970231"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.886035 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:09.886523 5724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.886682 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.886741 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "4ff4051b9feb40299a97d93f99970231"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.886783 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:09.891666 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:09.891894 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:09.892036 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:09.892227 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:09.892259 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.892282 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:09.892298 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.895169 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:46149
I20251026 09:10:09.895285 5787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:46149 every 8 connection(s)
I20251026 09:10:09.895622 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:09.897044 5792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:09.897156 5793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:09.897243 5795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:09.897395 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:09.897533 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:09.897572 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:09.897591 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469809897591 us; error 0 us; skew 500 ppm
I20251026 09:10:09.898365 3128 webserver.cc:492] Webserver started at http://127.3.14.2:44195/ using document root <none> and password file <none>
I20251026 09:10:09.898468 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:09.898519 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:09.898573 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:09.898900 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "f02a7a9c99c7434b83e7308d621cf582"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.899956 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 09:10:09.900470 5800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.900599 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.900650 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "f02a7a9c99c7434b83e7308d621cf582"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.900696 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:09.902750 5788 heartbeater.cc:344] Connected to a master server at 127.3.14.62:34649
I20251026 09:10:09.902822 5788 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:09.902940 5788 heartbeater.cc:507] Master 127.3.14.62:34649 requested a full tablet report, sending...
I20251026 09:10:09.903209 5659 ts_manager.cc:194] Registered new tserver with Master: 4ff4051b9feb40299a97d93f99970231 (127.3.14.1:46149)
I20251026 09:10:09.904064 5659 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47128
I20251026 09:10:09.924629 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:09.924868 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:09.925032 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:09.925235 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:09.925273 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.925297 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:09.925324 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.928462 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:40821
I20251026 09:10:09.928539 5863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:40821 every 8 connection(s)
I20251026 09:10:09.928894 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:09.930312 5868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:09.930414 5870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:09.930598 5867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:09.930681 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:09.930823 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:09.930853 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:09.930881 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469809930881 us; error 0 us; skew 500 ppm
I20251026 09:10:09.931838 3128 webserver.cc:492] Webserver started at http://127.3.14.3:43753/ using document root <none> and password file <none>
I20251026 09:10:09.931968 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:09.932035 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:09.932083 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:09.932313 5864 heartbeater.cc:344] Connected to a master server at 127.3.14.62:34649
I20251026 09:10:09.932384 5864 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:09.932410 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "f14d1433349a432792b4280fed15825c"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.932511 5864 heartbeater.cc:507] Master 127.3.14.62:34649 requested a full tablet report, sending...
I20251026 09:10:09.932778 5659 ts_manager.cc:194] Registered new tserver with Master: f02a7a9c99c7434b83e7308d621cf582 (127.3.14.2:40821)
I20251026 09:10:09.933141 5659 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47142
I20251026 09:10:09.933653 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 09:10:09.934172 5875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.934338 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.934391 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "f14d1433349a432792b4280fed15825c"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.934442 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:09.945883 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:09.946192 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:09.946388 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:09.946588 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:09.946616 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.946645 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:09.946671 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.950210 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:41365
I20251026 09:10:09.950294 5938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:41365 every 8 connection(s)
I20251026 09:10:09.950671 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:09.952131 5943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:09.952219 5942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:09.952450 3128 server_base.cc:1047] running on GCE node
W20251026 09:10:09.953039 5945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:09.953256 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:09.953313 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:09.953336 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469809953336 us; error 0 us; skew 500 ppm
I20251026 09:10:09.954173 3128 webserver.cc:492] Webserver started at http://127.3.14.4:34185/ using document root <none> and password file <none>
I20251026 09:10:09.954309 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:09.954370 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:09.954428 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:09.954442 5939 heartbeater.cc:344] Connected to a master server at 127.3.14.62:34649
I20251026 09:10:09.954522 5939 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:09.954639 5939 heartbeater.cc:507] Master 127.3.14.62:34649 requested a full tablet report, sending...
I20251026 09:10:09.954802 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-3-root/instance:
uuid: "8a14c2ec02fd465496c795f49d96df5e"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.954905 5659 ts_manager.cc:194] Registered new tserver with Master: f14d1433349a432792b4280fed15825c (127.3.14.3:41365)
I20251026 09:10:09.955281 5659 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47146
I20251026 09:10:09.955942 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:09.956524 5950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.956667 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.956713 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-3-root
uuid: "8a14c2ec02fd465496c795f49d96df5e"
format_stamp: "Formatted at 2025-10-26 09:10:09 on dist-test-slave-w8k0"
I20251026 09:10:09.956770 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761469742341867-3128-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:09.987963 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:09.988314 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:09.988507 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:09.988739 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:09.988770 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.988809 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:09.988840 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:09.992285 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.4:38863
I20251026 09:10:09.992383 6013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.4:38863 every 8 connection(s)
I20251026 09:10:09.995986 6014 heartbeater.cc:344] Connected to a master server at 127.3.14.62:34649
I20251026 09:10:09.996071 6014 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:09.996198 6014 heartbeater.cc:507] Master 127.3.14.62:34649 requested a full tablet report, sending...
I20251026 09:10:09.996484 5659 ts_manager.cc:194] Registered new tserver with Master: 8a14c2ec02fd465496c795f49d96df5e (127.3.14.4:38863)
I20251026 09:10:09.996832 5659 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47148
I20251026 09:10:09.996953 3128 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.004471884s
I20251026 09:10:10.904728 5788 heartbeater.cc:499] Master 127.3.14.62:34649 was elected leader, sending a full tablet report...
I20251026 09:10:10.933801 5864 heartbeater.cc:499] Master 127.3.14.62:34649 was elected leader, sending a full tablet report...
I20251026 09:10:10.956131 5939 heartbeater.cc:499] Master 127.3.14.62:34649 was elected leader, sending a full tablet report...
I20251026 09:10:10.997454 6014 heartbeater.cc:499] Master 127.3.14.62:34649 was elected leader, sending a full tablet report...
I20251026 09:10:11.027582 3128 test_util.cc:276] Using random seed: 643722874
I20251026 09:10:11.034366 5659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42338:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 09:10:11.039886 5978 tablet_service.cc:1505] Processing CreateTablet for tablet 244407a6516a4331829e9fe6627619a8 (DEFAULT_TABLE table=test-workload [id=05c8bac167ea445c8c81500badddaf2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 09:10:11.040071 5978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244407a6516a4331829e9fe6627619a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:11.040287 5828 tablet_service.cc:1505] Processing CreateTablet for tablet 244407a6516a4331829e9fe6627619a8 (DEFAULT_TABLE table=test-workload [id=05c8bac167ea445c8c81500badddaf2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 09:10:11.040377 5828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244407a6516a4331829e9fe6627619a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:11.041893 6035 tablet_bootstrap.cc:492] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e: Bootstrap starting.
I20251026 09:10:11.042057 6036 tablet_bootstrap.cc:492] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582: Bootstrap starting.
I20251026 09:10:11.042608 6035 tablet_bootstrap.cc:654] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:11.042709 6036 tablet_bootstrap.cc:654] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:11.043308 5752 tablet_service.cc:1505] Processing CreateTablet for tablet 244407a6516a4331829e9fe6627619a8 (DEFAULT_TABLE table=test-workload [id=05c8bac167ea445c8c81500badddaf2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 09:10:11.043308 6035 tablet_bootstrap.cc:492] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e: No bootstrap required, opened a new log
I20251026 09:10:11.043448 6035 ts_tablet_manager.cc:1403] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:10:11.043478 5752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244407a6516a4331829e9fe6627619a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:11.043550 6036 tablet_bootstrap.cc:492] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582: No bootstrap required, opened a new log
I20251026 09:10:11.043593 6036 ts_tablet_manager.cc:1403] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:10:11.043643 6035 raft_consensus.cc:359] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } }
I20251026 09:10:11.043702 6035 raft_consensus.cc:385] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:11.043716 6035 raft_consensus.cc:740] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a14c2ec02fd465496c795f49d96df5e, State: Initialized, Role: FOLLOWER
I20251026 09:10:11.043747 6036 raft_consensus.cc:359] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } }
I20251026 09:10:11.043766 6035 consensus_queue.cc:260] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } }
I20251026 09:10:11.043803 6036 raft_consensus.cc:385] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:11.043825 6036 raft_consensus.cc:740] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02a7a9c99c7434b83e7308d621cf582, State: Initialized, Role: FOLLOWER
I20251026 09:10:11.043879 6036 consensus_queue.cc:260] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } }
I20251026 09:10:11.043946 6035 ts_tablet_manager.cc:1434] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:11.044047 6036 ts_tablet_manager.cc:1434] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:11.045229 6041 tablet_bootstrap.cc:492] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231: Bootstrap starting.
I20251026 09:10:11.045773 6041 tablet_bootstrap.cc:654] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:11.046360 6041 tablet_bootstrap.cc:492] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231: No bootstrap required, opened a new log
I20251026 09:10:11.046407 6041 ts_tablet_manager.cc:1403] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:11.046545 6041 raft_consensus.cc:359] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } }
I20251026 09:10:11.046602 6041 raft_consensus.cc:385] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:11.046617 6041 raft_consensus.cc:740] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff4051b9feb40299a97d93f99970231, State: Initialized, Role: FOLLOWER
I20251026 09:10:11.046663 6041 consensus_queue.cc:260] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } }
I20251026 09:10:11.046837 6041 ts_tablet_manager.cc:1434] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20251026 09:10:11.146191 5789 tablet.cc:2378] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 09:10:11.181399 6039 raft_consensus.cc:493] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:11.181519 6039 raft_consensus.cc:515] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } }
I20251026 09:10:11.181722 6039 leader_election.cc:290] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ff4051b9feb40299a97d93f99970231 (127.3.14.1:46149), f02a7a9c99c7434b83e7308d621cf582 (127.3.14.2:40821)
I20251026 09:10:11.184747 5836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244407a6516a4331829e9fe6627619a8" candidate_uuid: "8a14c2ec02fd465496c795f49d96df5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02a7a9c99c7434b83e7308d621cf582" is_pre_election: true
I20251026 09:10:11.184765 5762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244407a6516a4331829e9fe6627619a8" candidate_uuid: "8a14c2ec02fd465496c795f49d96df5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff4051b9feb40299a97d93f99970231" is_pre_election: true
I20251026 09:10:11.184877 5836 raft_consensus.cc:2468] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a14c2ec02fd465496c795f49d96df5e in term 0.
I20251026 09:10:11.184955 5762 raft_consensus.cc:2468] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a14c2ec02fd465496c795f49d96df5e in term 0.
I20251026 09:10:11.185122 5951 leader_election.cc:304] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a14c2ec02fd465496c795f49d96df5e, f02a7a9c99c7434b83e7308d621cf582; no voters:
I20251026 09:10:11.185271 6039 raft_consensus.cc:2804] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:11.185335 6039 raft_consensus.cc:493] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:11.185362 6039 raft_consensus.cc:3060] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:11.186103 6039 raft_consensus.cc:515] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } }
I20251026 09:10:11.186269 6039 leader_election.cc:290] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [CANDIDATE]: Term 1 election: Requested vote from peers 4ff4051b9feb40299a97d93f99970231 (127.3.14.1:46149), f02a7a9c99c7434b83e7308d621cf582 (127.3.14.2:40821)
I20251026 09:10:11.186444 5836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244407a6516a4331829e9fe6627619a8" candidate_uuid: "8a14c2ec02fd465496c795f49d96df5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02a7a9c99c7434b83e7308d621cf582"
I20251026 09:10:11.186477 5762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "244407a6516a4331829e9fe6627619a8" candidate_uuid: "8a14c2ec02fd465496c795f49d96df5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff4051b9feb40299a97d93f99970231"
I20251026 09:10:11.186517 5836 raft_consensus.cc:3060] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:11.186532 5762 raft_consensus.cc:3060] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:11.187258 5762 raft_consensus.cc:2468] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a14c2ec02fd465496c795f49d96df5e in term 1.
I20251026 09:10:11.187357 5836 raft_consensus.cc:2468] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a14c2ec02fd465496c795f49d96df5e in term 1.
I20251026 09:10:11.187542 5953 leader_election.cc:304] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ff4051b9feb40299a97d93f99970231, 8a14c2ec02fd465496c795f49d96df5e, f02a7a9c99c7434b83e7308d621cf582; no voters:
I20251026 09:10:11.187650 6039 raft_consensus.cc:2804] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:11.187812 6039 raft_consensus.cc:697] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 1 LEADER]: Becoming Leader. State: Replica: 8a14c2ec02fd465496c795f49d96df5e, State: Running, Role: LEADER
I20251026 09:10:11.187880 6039 consensus_queue.cc:237] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } }
I20251026 09:10:11.188493 5658 catalog_manager.cc:5649] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a14c2ec02fd465496c795f49d96df5e (127.3.14.4). New cstate: current_term: 1 leader_uuid: "8a14c2ec02fd465496c795f49d96df5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a14c2ec02fd465496c795f49d96df5e" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 38863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:11.603049 6039 consensus_queue.cc:1048] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02a7a9c99c7434b83e7308d621cf582" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:11.604759 6052 consensus_queue.cc:1048] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff4051b9feb40299a97d93f99970231" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:11.866288 5713 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:11.866451 5713 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:11.866479 5713 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:13.866657 5713 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:13.866819 5713 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:13.866842 5713 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:15.867017 5713 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:15.867194 5713 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:15.867230 5713 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:17.867391 5713 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:17.867540 5713 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:17.867574 5713 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:19.867723 5713 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:19.867869 5713 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:19.867904 5713 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:21.868091 5713 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:21.868245 5713 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:21.868287 5713 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:22.270685 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:10:22.273468 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:22.273630 3128 tablet_replica.cc:333] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231: stopping tablet replica
I20251026 09:10:22.273698 3128 raft_consensus.cc:2243] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:22.273759 3128 raft_consensus.cc:2272] T 244407a6516a4331829e9fe6627619a8 P 4ff4051b9feb40299a97d93f99970231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:22.285705 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:10:22.286674 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:10:22.289587 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:22.289749 3128 tablet_replica.cc:333] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582: stopping tablet replica
I20251026 09:10:22.289814 3128 raft_consensus.cc:2243] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:22.289882 3128 raft_consensus.cc:2272] T 244407a6516a4331829e9fe6627619a8 P f02a7a9c99c7434b83e7308d621cf582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:22.301801 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:10:22.302672 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:10:22.305361 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:22.316922 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:10:22.317838 3128 tablet_server.cc:178] TabletServer@127.3.14.4:0 shutting down...
I20251026 09:10:22.320443 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:22.320611 3128 tablet_replica.cc:333] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e: stopping tablet replica
I20251026 09:10:22.320695 3128 raft_consensus.cc:2243] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:10:22.320772 3128 raft_consensus.cc:2272] T 244407a6516a4331829e9fe6627619a8 P 8a14c2ec02fd465496c795f49d96df5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:22.332551 3128 tablet_server.cc:195] TabletServer@127.3.14.4:0 shutdown complete.
I20251026 09:10:22.333500 3128 master.cc:561] Master@127.3.14.62:34649 shutting down...
I20251026 09:10:22.336019 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:10:22.336115 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:22.336138 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 18a6444240cf41bf9311ef8ec754b494: stopping tablet replica
I20251026 09:10:22.348100 3128 master.cc:583] Master@127.3.14.62:34649 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12514 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20251026 09:10:22.353255 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:39063
I20251026 09:10:22.353430 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:22.355191 6059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:22.355149 6056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:22.355119 6057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:22.355301 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:22.356002 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:22.356045 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:22.356065 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469822356064 us; error 0 us; skew 500 ppm
I20251026 09:10:22.356668 3128 webserver.cc:492] Webserver started at http://127.3.14.62:35085/ using document root <none> and password file <none>
I20251026 09:10:22.356755 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:22.356805 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:22.356859 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:22.357146 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "3b31948654cb464daeb72a5681237be0"
format_stamp: "Formatted at 2025-10-26 09:10:22 on dist-test-slave-w8k0"
I20251026 09:10:22.358111 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:22.358565 6064 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.358691 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.358754 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "3b31948654cb464daeb72a5681237be0"
format_stamp: "Formatted at 2025-10-26 09:10:22 on dist-test-slave-w8k0"
I20251026 09:10:22.358790 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:22.386128 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:22.386344 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:22.388976 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:39063
I20251026 09:10:22.391398 6116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:39063 every 8 connection(s)
I20251026 09:10:22.391527 6117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:22.392583 6117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0: Bootstrap starting.
I20251026 09:10:22.392894 6117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:22.393390 6117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0: No bootstrap required, opened a new log
I20251026 09:10:22.393508 6117 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b31948654cb464daeb72a5681237be0" member_type: VOTER }
I20251026 09:10:22.393572 6117 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:22.393586 6117 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b31948654cb464daeb72a5681237be0, State: Initialized, Role: FOLLOWER
I20251026 09:10:22.393630 6117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b31948654cb464daeb72a5681237be0" member_type: VOTER }
I20251026 09:10:22.393661 6117 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:10:22.393682 6117 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:10:22.393707 6117 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:22.394222 6117 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b31948654cb464daeb72a5681237be0" member_type: VOTER }
I20251026 09:10:22.394280 6117 leader_election.cc:304] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b31948654cb464daeb72a5681237be0; no voters:
I20251026 09:10:22.394395 6117 leader_election.cc:290] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:10:22.394455 6120 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:22.394551 6117 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:10:22.394618 6120 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 1 LEADER]: Becoming Leader. State: Replica: 3b31948654cb464daeb72a5681237be0, State: Running, Role: LEADER
I20251026 09:10:22.394683 6120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b31948654cb464daeb72a5681237be0" member_type: VOTER }
I20251026 09:10:22.394917 6121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b31948654cb464daeb72a5681237be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b31948654cb464daeb72a5681237be0" member_type: VOTER } }
I20251026 09:10:22.394941 6122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b31948654cb464daeb72a5681237be0. Latest consensus state: current_term: 1 leader_uuid: "3b31948654cb464daeb72a5681237be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b31948654cb464daeb72a5681237be0" member_type: VOTER } }
I20251026 09:10:22.394987 6121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [sys.catalog]: This master's current role is: LEADER
I20251026 09:10:22.395011 6122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [sys.catalog]: This master's current role is: LEADER
I20251026 09:10:22.395248 6125 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:10:22.395453 6125 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:10:22.396044 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:10:22.396052 6125 catalog_manager.cc:1357] Generated new cluster ID: ca7716a8a4364168bc6363f1ea5fa544
I20251026 09:10:22.396119 6125 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:10:22.404851 6125 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:10:22.405498 6125 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:10:22.413120 6125 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0: Generated new TSK 0
I20251026 09:10:22.413234 6125 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:10:22.427951 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:22.429534 6139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:22.429682 6138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:22.429742 3128 server_base.cc:1047] running on GCE node
W20251026 09:10:22.429762 6141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:22.430016 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:22.430056 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:22.430078 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469822430077 us; error 0 us; skew 500 ppm
I20251026 09:10:22.430758 3128 webserver.cc:492] Webserver started at http://127.3.14.1:40827/ using document root <none> and password file <none>
I20251026 09:10:22.430859 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:22.430907 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:22.430963 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:22.431259 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "bb78c84730be4ba093e042b17e2e1891"
format_stamp: "Formatted at 2025-10-26 09:10:22 on dist-test-slave-w8k0"
I20251026 09:10:22.432299 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:22.432983 6146 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.433173 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 09:10:22.433231 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "bb78c84730be4ba093e042b17e2e1891"
format_stamp: "Formatted at 2025-10-26 09:10:22 on dist-test-slave-w8k0"
I20251026 09:10:22.433295 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:22.449329 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:22.449635 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:22.449826 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:22.450037 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:22.450065 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.450103 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:22.450129 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.453109 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:38549
I20251026 09:10:22.453156 6209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:38549 every 8 connection(s)
I20251026 09:10:22.453519 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:22.454998 6214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:22.454967 6215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:22.454945 6217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:22.455202 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:22.455462 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:22.455518 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:22.455550 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469822455550 us; error 0 us; skew 500 ppm
I20251026 09:10:22.456223 3128 webserver.cc:492] Webserver started at http://127.3.14.2:40181/ using document root <none> and password file <none>
I20251026 09:10:22.456316 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:22.456365 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:22.456425 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:22.456761 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "c5f43c3254cf466cb131a972fa941792"
format_stamp: "Formatted at 2025-10-26 09:10:22 on dist-test-slave-w8k0"
I20251026 09:10:22.457700 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:22.458204 6222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.458355 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.458403 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "c5f43c3254cf466cb131a972fa941792"
format_stamp: "Formatted at 2025-10-26 09:10:22 on dist-test-slave-w8k0"
I20251026 09:10:22.458444 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:22.462141 6210 heartbeater.cc:344] Connected to a master server at 127.3.14.62:39063
I20251026 09:10:22.462204 6210 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:22.462318 6210 heartbeater.cc:507] Master 127.3.14.62:39063 requested a full tablet report, sending...
I20251026 09:10:22.462538 6081 ts_manager.cc:194] Registered new tserver with Master: bb78c84730be4ba093e042b17e2e1891 (127.3.14.1:38549)
I20251026 09:10:22.462889 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:22.463099 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:22.463119 6081 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59298
I20251026 09:10:22.463222 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:22.463470 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:22.463500 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.463532 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:22.463559 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.466439 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:35649
I20251026 09:10:22.466487 6285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:35649 every 8 connection(s)
I20251026 09:10:22.466846 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:22.468163 6290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:22.468334 6292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:22.468357 6289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:22.468595 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:22.468724 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:22.468757 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:22.468779 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469822468779 us; error 0 us; skew 500 ppm
I20251026 09:10:22.469522 3128 webserver.cc:492] Webserver started at http://127.3.14.3:34083/ using document root <none> and password file <none>
I20251026 09:10:22.469622 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:22.469678 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:22.469728 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:22.470058 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "78a1c8d14803425186de71dc0610615a"
format_stamp: "Formatted at 2025-10-26 09:10:22 on dist-test-slave-w8k0"
I20251026 09:10:22.470058 6286 heartbeater.cc:344] Connected to a master server at 127.3.14.62:39063
I20251026 09:10:22.470161 6286 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:22.470288 6286 heartbeater.cc:507] Master 127.3.14.62:39063 requested a full tablet report, sending...
I20251026 09:10:22.470516 6081 ts_manager.cc:194] Registered new tserver with Master: c5f43c3254cf466cb131a972fa941792 (127.3.14.2:35649)
I20251026 09:10:22.470853 6081 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59306
I20251026 09:10:22.471113 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:22.471695 6297 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.471848 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.471894 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "78a1c8d14803425186de71dc0610615a"
format_stamp: "Formatted at 2025-10-26 09:10:22 on dist-test-slave-w8k0"
I20251026 09:10:22.471944 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:22.491770 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:22.492188 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:22.492410 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:22.492623 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:22.492653 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.492690 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:22.492722 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:22.496192 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:45783
I20251026 09:10:22.496269 6360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:45783 every 8 connection(s)
I20251026 09:10:22.499413 6361 heartbeater.cc:344] Connected to a master server at 127.3.14.62:39063
I20251026 09:10:22.499492 6361 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:22.499599 6361 heartbeater.cc:507] Master 127.3.14.62:39063 requested a full tablet report, sending...
I20251026 09:10:22.499850 6081 ts_manager.cc:194] Registered new tserver with Master: 78a1c8d14803425186de71dc0610615a (127.3.14.3:45783)
I20251026 09:10:22.500209 6081 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59318
I20251026 09:10:22.500795 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00438926s
I20251026 09:10:23.463696 6210 heartbeater.cc:499] Master 127.3.14.62:39063 was elected leader, sending a full tablet report...
I20251026 09:10:23.471412 6286 heartbeater.cc:499] Master 127.3.14.62:39063 was elected leader, sending a full tablet report...
I20251026 09:10:23.500732 6361 heartbeater.cc:499] Master 127.3.14.62:39063 was elected leader, sending a full tablet report...
I20251026 09:10:23.529332 3128 test_util.cc:276] Using random seed: 656224625
I20251026 09:10:23.534277 6081 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59334:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:10:23.534453 6081 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:10:23.539480 6172 tablet_service.cc:1505] Processing CreateTablet for tablet d7384c610b93468bb1974194f6e08ef1 (DEFAULT_TABLE table=test-workload [id=066533a453ba4d4b96678be66dd6b9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 09:10:23.539664 6172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7384c610b93468bb1974194f6e08ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:23.540205 6325 tablet_service.cc:1505] Processing CreateTablet for tablet d7384c610b93468bb1974194f6e08ef1 (DEFAULT_TABLE table=test-workload [id=066533a453ba4d4b96678be66dd6b9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 09:10:23.540310 6325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7384c610b93468bb1974194f6e08ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:23.541038 6382 tablet_bootstrap.cc:492] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891: Bootstrap starting.
I20251026 09:10:23.541154 6250 tablet_service.cc:1505] Processing CreateTablet for tablet d7384c610b93468bb1974194f6e08ef1 (DEFAULT_TABLE table=test-workload [id=066533a453ba4d4b96678be66dd6b9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 09:10:23.541268 6250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7384c610b93468bb1974194f6e08ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:23.541538 6382 tablet_bootstrap.cc:654] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:23.542375 6385 tablet_bootstrap.cc:492] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792: Bootstrap starting.
I20251026 09:10:23.542459 6383 tablet_bootstrap.cc:492] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a: Bootstrap starting.
I20251026 09:10:23.542815 6385 tablet_bootstrap.cc:654] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:23.542891 6383 tablet_bootstrap.cc:654] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:23.543401 6385 tablet_bootstrap.cc:492] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792: No bootstrap required, opened a new log
I20251026 09:10:23.543416 6383 tablet_bootstrap.cc:492] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a: No bootstrap required, opened a new log
I20251026 09:10:23.543401 6382 tablet_bootstrap.cc:492] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891: No bootstrap required, opened a new log
I20251026 09:10:23.543537 6382 ts_tablet_manager.cc:1403] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 09:10:23.543476 6385 ts_tablet_manager.cc:1403] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:23.543529 6383 ts_tablet_manager.cc:1403] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:23.543802 6385 raft_consensus.cc:359] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } }
I20251026 09:10:23.543839 6383 raft_consensus.cc:359] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } }
I20251026 09:10:23.543802 6382 raft_consensus.cc:359] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } }
I20251026 09:10:23.543928 6383 raft_consensus.cc:385] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:23.543895 6385 raft_consensus.cc:385] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:23.543953 6382 raft_consensus.cc:385] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:23.543999 6382 raft_consensus.cc:740] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb78c84730be4ba093e042b17e2e1891, State: Initialized, Role: FOLLOWER
I20251026 09:10:23.543953 6383 raft_consensus.cc:740] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a1c8d14803425186de71dc0610615a, State: Initialized, Role: FOLLOWER
I20251026 09:10:23.543972 6385 raft_consensus.cc:740] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f43c3254cf466cb131a972fa941792, State: Initialized, Role: FOLLOWER
I20251026 09:10:23.544064 6382 consensus_queue.cc:260] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } }
I20251026 09:10:23.544083 6383 consensus_queue.cc:260] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } }
I20251026 09:10:23.544106 6385 consensus_queue.cc:260] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } }
I20251026 09:10:23.544230 6382 ts_tablet_manager.cc:1434] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:23.544230 6383 ts_tablet_manager.cc:1434] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:10:23.544355 6385 ts_tablet_manager.cc:1434] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:23.666424 6389 raft_consensus.cc:493] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:23.666543 6389 raft_consensus.cc:515] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } }
I20251026 09:10:23.666762 6389 leader_election.cc:290] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb78c84730be4ba093e042b17e2e1891 (127.3.14.1:38549), c5f43c3254cf466cb131a972fa941792 (127.3.14.2:35649)
I20251026 09:10:23.669504 6184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7384c610b93468bb1974194f6e08ef1" candidate_uuid: "78a1c8d14803425186de71dc0610615a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb78c84730be4ba093e042b17e2e1891" is_pre_election: true
I20251026 09:10:23.669607 6184 raft_consensus.cc:2468] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78a1c8d14803425186de71dc0610615a in term 0.
I20251026 09:10:23.669512 6260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7384c610b93468bb1974194f6e08ef1" candidate_uuid: "78a1c8d14803425186de71dc0610615a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f43c3254cf466cb131a972fa941792" is_pre_election: true
I20251026 09:10:23.669654 6260 raft_consensus.cc:2468] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78a1c8d14803425186de71dc0610615a in term 0.
I20251026 09:10:23.669749 6299 leader_election.cc:304] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78a1c8d14803425186de71dc0610615a, bb78c84730be4ba093e042b17e2e1891; no voters:
I20251026 09:10:23.669837 6389 raft_consensus.cc:2804] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:23.669884 6389 raft_consensus.cc:493] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:23.669900 6389 raft_consensus.cc:3060] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:23.670540 6389 raft_consensus.cc:515] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } }
I20251026 09:10:23.670660 6389 leader_election.cc:290] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [CANDIDATE]: Term 1 election: Requested vote from peers bb78c84730be4ba093e042b17e2e1891 (127.3.14.1:38549), c5f43c3254cf466cb131a972fa941792 (127.3.14.2:35649)
I20251026 09:10:23.670809 6260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7384c610b93468bb1974194f6e08ef1" candidate_uuid: "78a1c8d14803425186de71dc0610615a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f43c3254cf466cb131a972fa941792"
I20251026 09:10:23.670807 6184 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7384c610b93468bb1974194f6e08ef1" candidate_uuid: "78a1c8d14803425186de71dc0610615a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb78c84730be4ba093e042b17e2e1891"
I20251026 09:10:23.670869 6260 raft_consensus.cc:3060] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:23.670905 6184 raft_consensus.cc:3060] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:23.671605 6260 raft_consensus.cc:2468] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78a1c8d14803425186de71dc0610615a in term 1.
I20251026 09:10:23.671610 6184 raft_consensus.cc:2468] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78a1c8d14803425186de71dc0610615a in term 1.
I20251026 09:10:23.671767 6298 leader_election.cc:304] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78a1c8d14803425186de71dc0610615a, c5f43c3254cf466cb131a972fa941792; no voters:
I20251026 09:10:23.671851 6389 raft_consensus.cc:2804] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:23.672008 6389 raft_consensus.cc:697] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 1 LEADER]: Becoming Leader. State: Replica: 78a1c8d14803425186de71dc0610615a, State: Running, Role: LEADER
I20251026 09:10:23.672056 6389 consensus_queue.cc:237] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } }
I20251026 09:10:23.672747 6079 catalog_manager.cc:5649] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a reported cstate change: term changed from 0 to 1, leader changed from <none> to 78a1c8d14803425186de71dc0610615a (127.3.14.3). New cstate: current_term: 1 leader_uuid: "78a1c8d14803425186de71dc0610615a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78a1c8d14803425186de71dc0610615a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45783 } health_report { overall_health: HEALTHY } } }
I20251026 09:10:24.155896 6389 consensus_queue.cc:1048] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f43c3254cf466cb131a972fa941792" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 35649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:24.157338 6398 consensus_queue.cc:1048] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb78c84730be4ba093e042b17e2e1891" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:24.395887 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:24.396083 6136 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:24.396163 6136 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:24.396200 6136 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:25.396468 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:26.396384 6136 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:26.396514 6136 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:26.396534 6136 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:26.396827 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:27.397234 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:28.396682 6136 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:28.396816 6136 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:28.396842 6136 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:28.397586 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:29.397927 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:30.396998 6136 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:30.397135 6136 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:30.397159 6136 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:30.398283 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:31.398681 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:32.397378 6136 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:32.397531 6136 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:32.397557 6136 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:32.399089 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:33.399499 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:34.397722 6136 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:34.397871 6136 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:10:34.397894 6136 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:34.400014 6130 placement_policy_util.cc:407] tablet d7384c610b93468bb1974194f6e08ef1: detected majority of replicas (2 of 3) at location L0
I20251026 09:10:34.733994 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:10:34.736536 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:34.736690 3128 tablet_replica.cc:333] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891: stopping tablet replica
I20251026 09:10:34.736764 3128 raft_consensus.cc:2243] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:34.736821 3128 raft_consensus.cc:2272] T d7384c610b93468bb1974194f6e08ef1 P bb78c84730be4ba093e042b17e2e1891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:34.748937 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:10:34.749931 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:10:34.752954 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:34.753090 3128 tablet_replica.cc:333] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792: stopping tablet replica
I20251026 09:10:34.753154 3128 raft_consensus.cc:2243] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:34.753223 3128 raft_consensus.cc:2272] T d7384c610b93468bb1974194f6e08ef1 P c5f43c3254cf466cb131a972fa941792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:34.764966 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:10:34.765856 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
W20251026 09:10:34.768406 6298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.14.2:35649: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20251026 09:10:34.768896 6298 consensus_peers.cc:597] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a -> Peer c5f43c3254cf466cb131a972fa941792 (127.3.14.2:35649): Couldn't send request to peer c5f43c3254cf466cb131a972fa941792. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.2:35649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:34.769348 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:10:34.769469 3128 tablet_replica.cc:333] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a: stopping tablet replica
I20251026 09:10:34.769528 3128 raft_consensus.cc:2243] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:10:34.769608 3128 raft_consensus.cc:2272] T d7384c610b93468bb1974194f6e08ef1 P 78a1c8d14803425186de71dc0610615a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:34.781419 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:10:34.782404 3128 master.cc:561] Master@127.3.14.62:39063 shutting down...
I20251026 09:10:34.784647 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:10:34.784751 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:34.784773 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3b31948654cb464daeb72a5681237be0: stopping tablet replica
I20251026 09:10:34.796613 3128 master.cc:583] Master@127.3.14.62:39063 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12447 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20251026 09:10:34.801292 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:38937
I20251026 09:10:34.801474 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:34.802732 6405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:34.802788 6408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:34.802901 6406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:34.803027 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:34.803138 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:34.803179 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:34.803210 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469834803210 us; error 0 us; skew 500 ppm
I20251026 09:10:34.803839 3128 webserver.cc:492] Webserver started at http://127.3.14.62:44523/ using document root <none> and password file <none>
I20251026 09:10:34.803920 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:34.803980 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:34.804029 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:34.804358 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "0194cb86687941f2a46146ccf9f70022"
format_stamp: "Formatted at 2025-10-26 09:10:34 on dist-test-slave-w8k0"
I20251026 09:10:34.805380 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:34.805902 6413 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.806102 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 09:10:34.806151 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "0194cb86687941f2a46146ccf9f70022"
format_stamp: "Formatted at 2025-10-26 09:10:34 on dist-test-slave-w8k0"
I20251026 09:10:34.806216 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:34.816289 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:34.816543 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:34.819240 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:38937
I20251026 09:10:34.821141 6465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:38937 every 8 connection(s)
I20251026 09:10:34.821293 6466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:34.822368 6466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022: Bootstrap starting.
I20251026 09:10:34.822651 6466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:34.823151 6466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022: No bootstrap required, opened a new log
I20251026 09:10:34.823277 6466 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0194cb86687941f2a46146ccf9f70022" member_type: VOTER }
I20251026 09:10:34.823376 6466 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:34.823392 6466 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0194cb86687941f2a46146ccf9f70022, State: Initialized, Role: FOLLOWER
I20251026 09:10:34.823433 6466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0194cb86687941f2a46146ccf9f70022" member_type: VOTER }
I20251026 09:10:34.823477 6466 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:10:34.823498 6466 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:10:34.823530 6466 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:34.824055 6466 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0194cb86687941f2a46146ccf9f70022" member_type: VOTER }
I20251026 09:10:34.824115 6466 leader_election.cc:304] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0194cb86687941f2a46146ccf9f70022; no voters:
I20251026 09:10:34.824221 6466 leader_election.cc:290] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:10:34.824303 6469 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:34.824378 6466 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:10:34.824482 6469 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 1 LEADER]: Becoming Leader. State: Replica: 0194cb86687941f2a46146ccf9f70022, State: Running, Role: LEADER
I20251026 09:10:34.824544 6469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0194cb86687941f2a46146ccf9f70022" member_type: VOTER }
I20251026 09:10:34.824774 6470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0194cb86687941f2a46146ccf9f70022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0194cb86687941f2a46146ccf9f70022" member_type: VOTER } }
I20251026 09:10:34.824781 6471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0194cb86687941f2a46146ccf9f70022. Latest consensus state: current_term: 1 leader_uuid: "0194cb86687941f2a46146ccf9f70022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0194cb86687941f2a46146ccf9f70022" member_type: VOTER } }
I20251026 09:10:34.824854 6470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [sys.catalog]: This master's current role is: LEADER
I20251026 09:10:34.824860 6471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [sys.catalog]: This master's current role is: LEADER
I20251026 09:10:34.825012 6474 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:10:34.825189 6474 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:10:34.825815 6474 catalog_manager.cc:1357] Generated new cluster ID: 893f369434df452999374102d0428bd4
I20251026 09:10:34.825867 6474 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:10:34.825865 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:10:34.837234 6474 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:10:34.837688 6474 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:10:34.849941 6474 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022: Generated new TSK 0
I20251026 09:10:34.850080 6474 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:10:34.857998 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:34.859534 6487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:34.859635 6488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:34.859643 6490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:34.859822 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:34.860003 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:34.860044 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:34.860067 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469834860067 us; error 0 us; skew 500 ppm
I20251026 09:10:34.860766 3128 webserver.cc:492] Webserver started at http://127.3.14.1:37581/ using document root <none> and password file <none>
I20251026 09:10:34.860867 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:34.860930 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:34.860981 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:34.861275 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "3ed603c643ac44ca9908b171687d7d5c"
format_stamp: "Formatted at 2025-10-26 09:10:34 on dist-test-slave-w8k0"
I20251026 09:10:34.862213 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:34.862684 6495 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.862818 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.862864 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "3ed603c643ac44ca9908b171687d7d5c"
format_stamp: "Formatted at 2025-10-26 09:10:34 on dist-test-slave-w8k0"
I20251026 09:10:34.862951 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:34.890434 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:34.890710 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:34.890884 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:34.891113 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:34.891145 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.891180 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:34.891209 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.894150 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:33705
I20251026 09:10:34.894202 6558 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:33705 every 8 connection(s)
I20251026 09:10:34.894559 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:34.895998 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:34.896093 6564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:34.896229 3128 server_base.cc:1047] running on GCE node
W20251026 09:10:34.896111 6563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:34.897001 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:34.897058 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:34.897076 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469834897077 us; error 0 us; skew 500 ppm
I20251026 09:10:34.897899 3128 webserver.cc:492] Webserver started at http://127.3.14.2:39781/ using document root <none> and password file <none>
I20251026 09:10:34.898010 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:34.898070 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:34.898159 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:34.898639 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "cfcab08da3db40e1974902b787ef6361"
format_stamp: "Formatted at 2025-10-26 09:10:34 on dist-test-slave-w8k0"
I20251026 09:10:34.898712 6559 heartbeater.cc:344] Connected to a master server at 127.3.14.62:38937
I20251026 09:10:34.898770 6559 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:34.898924 6559 heartbeater.cc:507] Master 127.3.14.62:38937 requested a full tablet report, sending...
I20251026 09:10:34.899204 6430 ts_manager.cc:194] Registered new tserver with Master: 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705)
I20251026 09:10:34.899876 6430 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43290
I20251026 09:10:34.899899 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 09:10:34.900470 6571 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.900604 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 09:10:34.900651 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "cfcab08da3db40e1974902b787ef6361"
format_stamp: "Formatted at 2025-10-26 09:10:34 on dist-test-slave-w8k0"
I20251026 09:10:34.900708 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:34.916602 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:34.916930 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:34.917261 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:34.917501 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:34.917536 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.917572 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:34.917600 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.921010 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:42631
I20251026 09:10:34.921054 6634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:42631 every 8 connection(s)
I20251026 09:10:34.921483 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:34.923050 6639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:34.923065 6638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:34.923316 6641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:34.923463 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:34.923607 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:34.923656 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:34.923676 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469834923676 us; error 0 us; skew 500 ppm
I20251026 09:10:34.924599 3128 webserver.cc:492] Webserver started at http://127.3.14.3:42323/ using document root <none> and password file <none>
I20251026 09:10:34.924736 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:34.924803 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:34.924873 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:34.925280 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
format_stamp: "Formatted at 2025-10-26 09:10:34 on dist-test-slave-w8k0"
I20251026 09:10:34.925333 6635 heartbeater.cc:344] Connected to a master server at 127.3.14.62:38937
I20251026 09:10:34.925400 6635 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:34.925494 6635 heartbeater.cc:507] Master 127.3.14.62:38937 requested a full tablet report, sending...
I20251026 09:10:34.925798 6430 ts_manager.cc:194] Registered new tserver with Master: cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:34.926141 6430 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43304
I20251026 09:10:34.926527 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 09:10:34.927182 6646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.927402 3128 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 09:10:34.927465 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
format_stamp: "Formatted at 2025-10-26 09:10:34 on dist-test-slave-w8k0"
I20251026 09:10:34.927526 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:34.938865 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:34.939283 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:34.939533 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:34.939767 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:34.939800 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.939839 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:34.939859 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:34.943383 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:39689
I20251026 09:10:34.943495 6709 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:39689 every 8 connection(s)
I20251026 09:10:34.947207 6710 heartbeater.cc:344] Connected to a master server at 127.3.14.62:38937
I20251026 09:10:34.947290 6710 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:34.947438 6710 heartbeater.cc:507] Master 127.3.14.62:38937 requested a full tablet report, sending...
I20251026 09:10:34.947690 6430 ts_manager.cc:194] Registered new tserver with Master: 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:34.948046 6430 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43308
I20251026 09:10:34.948124 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00450668s
I20251026 09:10:35.900496 6559 heartbeater.cc:499] Master 127.3.14.62:38937 was elected leader, sending a full tablet report...
I20251026 09:10:35.926688 6635 heartbeater.cc:499] Master 127.3.14.62:38937 was elected leader, sending a full tablet report...
I20251026 09:10:35.948699 6710 heartbeater.cc:499] Master 127.3.14.62:38937 was elected leader, sending a full tablet report...
I20251026 09:10:35.973627 3128 test_util.cc:276] Using random seed: 668668919
I20251026 09:10:35.979597 6430 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43316:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:10:35.979789 6430 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:10:35.986313 6597 tablet_service.cc:1505] Processing CreateTablet for tablet ac1d836cf54d42c39eb2d8279fd17f90 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20251026 09:10:35.986375 6674 tablet_service.cc:1505] Processing CreateTablet for tablet e65f6e7109bc4eda8d83d836786c4d16 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION VALUES < 178956970
I20251026 09:10:35.986433 6598 tablet_service.cc:1505] Processing CreateTablet for tablet ff5ceda9af754029a9b5ac636a17ea70 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20251026 09:10:35.986512 6597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1d836cf54d42c39eb2d8279fd17f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.986526 6674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65f6e7109bc4eda8d83d836786c4d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.986588 6670 tablet_service.cc:1505] Processing CreateTablet for tablet 90072194423a4007b85cbc3f31889400 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20251026 09:10:35.986644 6595 tablet_service.cc:1505] Processing CreateTablet for tablet 90072194423a4007b85cbc3f31889400 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20251026 09:10:35.986714 6595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90072194423a4007b85cbc3f31889400. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.986768 6669 tablet_service.cc:1505] Processing CreateTablet for tablet ad759f01ba6c4154968854b4b9378651 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20251026 09:10:35.987093 6593 tablet_service.cc:1505] Processing CreateTablet for tablet 6459af4a4180422a8eccf12fc862499e (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20251026 09:10:35.987178 6593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6459af4a4180422a8eccf12fc862499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.988225 6665 tablet_service.cc:1505] Processing CreateTablet for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20251026 09:10:35.988273 6597 tablet_service.cc:1505] Processing CreateTablet for tablet e002d8e9fb004210be27d87c5a23f9c3 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20251026 09:10:35.988279 6674 tablet_service.cc:1505] Processing CreateTablet for tablet e002d8e9fb004210be27d87c5a23f9c3 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20251026 09:10:35.988380 6597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e002d8e9fb004210be27d87c5a23f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.988392 6666 tablet_service.cc:1505] Processing CreateTablet for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20251026 09:10:35.988528 6667 tablet_service.cc:1505] Processing CreateTablet for tablet 22a8d92b27b14f89a4728a8f50db932b (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20251026 09:10:35.988641 6668 tablet_service.cc:1505] Processing CreateTablet for tablet 6459af4a4180422a8eccf12fc862499e (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20251026 09:10:35.988692 6593 tablet_service.cc:1505] Processing CreateTablet for tablet f9723f81063e4f23bb2300f2c7bd50eb (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20251026 09:10:35.988765 6593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9723f81063e4f23bb2300f2c7bd50eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.988812 6671 tablet_service.cc:1505] Processing CreateTablet for tablet 21bd52ca927842ccab72fc70494128d0 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20251026 09:10:35.988945 6672 tablet_service.cc:1505] Processing CreateTablet for tablet ac1d836cf54d42c39eb2d8279fd17f90 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20251026 09:10:35.989082 6673 tablet_service.cc:1505] Processing CreateTablet for tablet ff5ceda9af754029a9b5ac636a17ea70 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20251026 09:10:35.989202 6590 tablet_service.cc:1505] Processing CreateTablet for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20251026 09:10:35.989279 6590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.990252 6731 tablet_bootstrap.cc:492] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:35.990702 6591 tablet_service.cc:1505] Processing CreateTablet for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20251026 09:10:35.990804 6731 tablet_bootstrap.cc:654] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:35.990869 6592 tablet_service.cc:1505] Processing CreateTablet for tablet 22a8d92b27b14f89a4728a8f50db932b (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20251026 09:10:35.991044 6523 tablet_service.cc:1505] Processing CreateTablet for tablet e65f6e7109bc4eda8d83d836786c4d16 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION VALUES < 178956970
I20251026 09:10:35.991132 6523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65f6e7109bc4eda8d83d836786c4d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.991406 6730 tablet_bootstrap.cc:492] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:35.991688 6731 tablet_bootstrap.cc:492] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:35.991740 6731 ts_tablet_manager.cc:1403] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:10:35.991950 6731 raft_consensus.cc:359] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:35.992034 6731 raft_consensus.cc:385] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:35.992061 6731 raft_consensus.cc:740] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:35.992151 6731 consensus_queue.cc:260] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:35.992372 6731 ts_tablet_manager.cc:1434] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:35.992453 6731 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:35.992884 6523 tablet_service.cc:1505] Processing CreateTablet for tablet e002d8e9fb004210be27d87c5a23f9c3 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20251026 09:10:35.992977 6523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e002d8e9fb004210be27d87c5a23f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.993012 6734 tablet_bootstrap.cc:492] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:35.993608 6734 tablet_bootstrap.cc:654] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:35.994179 6523 tablet_service.cc:1505] Processing CreateTablet for tablet f9723f81063e4f23bb2300f2c7bd50eb (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20251026 09:10:35.994289 6523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9723f81063e4f23bb2300f2c7bd50eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.987006 6594 tablet_service.cc:1505] Processing CreateTablet for tablet ad759f01ba6c4154968854b4b9378651 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20251026 09:10:35.995415 6522 tablet_service.cc:1505] Processing CreateTablet for tablet ff5ceda9af754029a9b5ac636a17ea70 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20251026 09:10:35.986708 6670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90072194423a4007b85cbc3f31889400. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.995513 6522 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5ceda9af754029a9b5ac636a17ea70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.995585 6668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6459af4a4180422a8eccf12fc862499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.996183 6730 tablet_bootstrap.cc:654] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:35.996906 6521 tablet_service.cc:1505] Processing CreateTablet for tablet ac1d836cf54d42c39eb2d8279fd17f90 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20251026 09:10:35.997010 6521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1d836cf54d42c39eb2d8279fd17f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.997108 6668 tablet_service.cc:1505] Processing CreateTablet for tablet f9723f81063e4f23bb2300f2c7bd50eb (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20251026 09:10:35.997203 6668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9723f81063e4f23bb2300f2c7bd50eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.998099 6674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e002d8e9fb004210be27d87c5a23f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.998310 6520 tablet_service.cc:1505] Processing CreateTablet for tablet 21bd52ca927842ccab72fc70494128d0 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20251026 09:10:35.998399 6666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb5819b36e244d4b0ab2c0e110e51e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.998469 6519 tablet_service.cc:1505] Processing CreateTablet for tablet 90072194423a4007b85cbc3f31889400 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20251026 09:10:35.998623 6518 tablet_service.cc:1505] Processing CreateTablet for tablet ad759f01ba6c4154968854b4b9378651 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20251026 09:10:35.998754 6517 tablet_service.cc:1505] Processing CreateTablet for tablet 6459af4a4180422a8eccf12fc862499e (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20251026 09:10:35.998890 6516 tablet_service.cc:1505] Processing CreateTablet for tablet 22a8d92b27b14f89a4728a8f50db932b (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20251026 09:10:35.999025 6515 tablet_service.cc:1505] Processing CreateTablet for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20251026 09:10:35.999162 6514 tablet_service.cc:1505] Processing CreateTablet for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20251026 09:10:35.999393 6669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad759f01ba6c4154968854b4b9378651. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.986397 6599 tablet_service.cc:1505] Processing CreateTablet for tablet e65f6e7109bc4eda8d83d836786c4d16 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION VALUES < 178956970
I20251026 09:10:35.986599 6596 tablet_service.cc:1505] Processing CreateTablet for tablet 21bd52ca927842ccab72fc70494128d0 (DEFAULT_TABLE table=test-workload [id=801ba6b94b4647619e2c65accdea7862]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20251026 09:10:36.001016 6673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5ceda9af754029a9b5ac636a17ea70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.002455 6672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1d836cf54d42c39eb2d8279fd17f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.003856 6667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22a8d92b27b14f89a4728a8f50db932b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.005179 6671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bd52ca927842ccab72fc70494128d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.998409 6520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bd52ca927842ccab72fc70494128d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.006675 6519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90072194423a4007b85cbc3f31889400. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.006682 6665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:35.990810 6591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb5819b36e244d4b0ab2c0e110e51e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.006871 6514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.008358 6515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb5819b36e244d4b0ab2c0e110e51e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.008522 6592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22a8d92b27b14f89a4728a8f50db932b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.008745 6598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5ceda9af754029a9b5ac636a17ea70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.009315 6734 tablet_bootstrap.cc:492] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.009369 6734 ts_tablet_manager.cc:1403] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20251026 09:10:36.009555 6734 raft_consensus.cc:359] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.009636 6734 raft_consensus.cc:385] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.009658 6734 raft_consensus.cc:740] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.009784 6599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65f6e7109bc4eda8d83d836786c4d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.009853 6734 consensus_queue.cc:260] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.010057 6734 ts_tablet_manager.cc:1434] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.010262 6734 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.010454 6594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad759f01ba6c4154968854b4b9378651. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.010632 6517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6459af4a4180422a8eccf12fc862499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.010747 6734 tablet_bootstrap.cc:654] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.011767 6596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bd52ca927842ccab72fc70494128d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.012148 6516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22a8d92b27b14f89a4728a8f50db932b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.012357 6730 tablet_bootstrap.cc:492] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.012403 6730 ts_tablet_manager.cc:1403] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20251026 09:10:36.012570 6730 raft_consensus.cc:359] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.012636 6730 raft_consensus.cc:385] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.012657 6730 raft_consensus.cc:740] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.012723 6730 consensus_queue.cc:260] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.012897 6518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad759f01ba6c4154968854b4b9378651. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:36.012920 6730 ts_tablet_manager.cc:1434] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:36.013223 6730 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.013744 6730 tablet_bootstrap.cc:654] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.014506 6730 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.014519 6731 tablet_bootstrap.cc:654] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.014554 6730 ts_tablet_manager.cc:1403] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.014714 6730 raft_consensus.cc:359] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.014773 6730 raft_consensus.cc:385] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.014788 6730 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.014848 6730 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.014942 6730 ts_tablet_manager.cc:1434] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.015097 6734 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.015141 6730 tablet_bootstrap.cc:492] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.015166 6734 ts_tablet_manager.cc:1403] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20251026 09:10:36.015178 6731 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.015239 6731 ts_tablet_manager.cc:1403] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20251026 09:10:36.015424 6734 raft_consensus.cc:359] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.015450 6731 raft_consensus.cc:359] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.015498 6734 raft_consensus.cc:385] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.015527 6734 raft_consensus.cc:740] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.015516 6731 raft_consensus.cc:385] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.015601 6730 tablet_bootstrap.cc:654] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.015609 6734 consensus_queue.cc:260] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.015756 6734 ts_tablet_manager.cc:1434] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.015858 6734 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.016291 6734 tablet_bootstrap.cc:654] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.016306 6730 tablet_bootstrap.cc:492] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.016356 6730 ts_tablet_manager.cc:1403] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.016515 6730 raft_consensus.cc:359] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.016587 6730 raft_consensus.cc:385] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.015596 6731 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.016682 6731 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.016825 6731 ts_tablet_manager.cc:1434] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 09:10:36.016893 6731 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.016920 6734 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.016974 6734 ts_tablet_manager.cc:1403] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.017127 6734 raft_consensus.cc:359] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.017196 6734 raft_consensus.cc:385] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.017220 6734 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.017279 6734 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.017323 6731 tablet_bootstrap.cc:654] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.017375 6734 ts_tablet_manager.cc:1434] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:36.017454 6734 tablet_bootstrap.cc:492] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.016610 6730 raft_consensus.cc:740] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.017721 6730 consensus_queue.cc:260] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.017869 6730 ts_tablet_manager.cc:1434] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.017880 6734 tablet_bootstrap.cc:654] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.017975 6730 tablet_bootstrap.cc:492] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.018409 6730 tablet_bootstrap.cc:654] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.018599 6731 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.018647 6731 ts_tablet_manager.cc:1403] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:10:36.018806 6731 raft_consensus.cc:359] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.018891 6731 raft_consensus.cc:385] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.018919 6731 raft_consensus.cc:740] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.018970 6734 tablet_bootstrap.cc:492] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.018996 6731 consensus_queue.cc:260] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.019018 6734 ts_tablet_manager.cc:1403] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:10:36.019057 6730 tablet_bootstrap.cc:492] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.019104 6730 ts_tablet_manager.cc:1403] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.019109 6731 ts_tablet_manager.cc:1434] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:36.019212 6734 raft_consensus.cc:359] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.019285 6734 raft_consensus.cc:385] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.019313 6734 raft_consensus.cc:740] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.019373 6730 raft_consensus.cc:359] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.019381 6734 consensus_queue.cc:260] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.019455 6730 raft_consensus.cc:385] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.019483 6730 raft_consensus.cc:740] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.019523 6734 ts_tablet_manager.cc:1434] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.019553 6730 consensus_queue.cc:260] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.019590 6734 tablet_bootstrap.cc:492] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.019659 6730 ts_tablet_manager.cc:1434] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:36.019716 6730 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.020016 6734 tablet_bootstrap.cc:654] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.020159 6730 tablet_bootstrap.cc:654] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.020155 6731 tablet_bootstrap.cc:492] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.020659 6731 tablet_bootstrap.cc:654] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.020732 6734 tablet_bootstrap.cc:492] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.020771 6730 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.020783 6734 ts_tablet_manager.cc:1403] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.020839 6730 ts_tablet_manager.cc:1403] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.020957 6734 raft_consensus.cc:359] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.020983 6730 raft_consensus.cc:359] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.021027 6734 raft_consensus.cc:385] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.021041 6730 raft_consensus.cc:385] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.021076 6730 raft_consensus.cc:740] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.021052 6734 raft_consensus.cc:740] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.021150 6730 consensus_queue.cc:260] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.021173 6734 consensus_queue.cc:260] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.021251 6730 ts_tablet_manager.cc:1434] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:36.021334 6730 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.021479 6731 tablet_bootstrap.cc:492] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.021251 6734 ts_tablet_manager.cc:1434] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.021528 6731 ts_tablet_manager.cc:1403] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.021574 6734 tablet_bootstrap.cc:492] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.021675 6731 raft_consensus.cc:359] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.021737 6731 raft_consensus.cc:385] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.021770 6731 raft_consensus.cc:740] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.021785 6730 tablet_bootstrap.cc:654] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.021835 6731 consensus_queue.cc:260] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.021960 6731 ts_tablet_manager.cc:1434] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:36.022007 6734 tablet_bootstrap.cc:654] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.022071 6731 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.022549 6730 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.022599 6730 ts_tablet_manager.cc:1403] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.022747 6730 raft_consensus.cc:359] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.022826 6730 raft_consensus.cc:385] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.022851 6730 raft_consensus.cc:740] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.022886 6734 tablet_bootstrap.cc:492] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.022894 6730 consensus_queue.cc:260] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.023046 6730 ts_tablet_manager.cc:1434] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.023109 6731 tablet_bootstrap.cc:654] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.023126 6730 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.022943 6734 ts_tablet_manager.cc:1403] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.023561 6734 raft_consensus.cc:359] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.023614 6730 tablet_bootstrap.cc:654] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.023631 6734 raft_consensus.cc:385] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.023658 6734 raft_consensus.cc:740] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.023700 6731 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.023763 6731 ts_tablet_manager.cc:1403] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:10:36.023702 6734 consensus_queue.cc:260] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.023865 6734 ts_tablet_manager.cc:1434] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.023919 6734 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.023965 6731 raft_consensus.cc:359] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.024039 6731 raft_consensus.cc:385] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.024066 6731 raft_consensus.cc:740] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.024120 6731 consensus_queue.cc:260] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.024187 6730 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.024221 6731 ts_tablet_manager.cc:1434] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:10:36.024232 6730 ts_tablet_manager.cc:1403] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.024304 6731 tablet_bootstrap.cc:492] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.024350 6734 tablet_bootstrap.cc:654] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.024425 6730 raft_consensus.cc:359] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.024499 6730 raft_consensus.cc:385] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.024525 6730 raft_consensus.cc:740] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.024586 6730 consensus_queue.cc:260] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.024677 6730 ts_tablet_manager.cc:1434] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.024768 6731 tablet_bootstrap.cc:654] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.024768 6730 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.025048 6734 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.025090 6734 ts_tablet_manager.cc:1403] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.025233 6734 raft_consensus.cc:359] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.025293 6734 raft_consensus.cc:385] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.025329 6734 raft_consensus.cc:740] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.025370 6734 consensus_queue.cc:260] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.025413 6731 tablet_bootstrap.cc:492] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.025480 6731 ts_tablet_manager.cc:1403] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.025302 6730 tablet_bootstrap.cc:654] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.025640 6731 raft_consensus.cc:359] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.025722 6731 raft_consensus.cc:385] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.025749 6731 raft_consensus.cc:740] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.025792 6731 consensus_queue.cc:260] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.025893 6731 ts_tablet_manager.cc:1434] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.025482 6734 ts_tablet_manager.cc:1434] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:36.026054 6731 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.026054 6734 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.026540 6734 tablet_bootstrap.cc:654] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.026540 6731 tablet_bootstrap.cc:654] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.026804 6730 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.026865 6730 ts_tablet_manager.cc:1403] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:10:36.027040 6730 raft_consensus.cc:359] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.027129 6730 raft_consensus.cc:385] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.027163 6730 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.027189 6734 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.027215 6730 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.027238 6734 ts_tablet_manager.cc:1403] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.027278 6731 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.027361 6730 ts_tablet_manager.cc:1434] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:10:36.027383 6731 ts_tablet_manager.cc:1403] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.027420 6730 tablet_bootstrap.cc:492] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.027479 6734 raft_consensus.cc:359] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.027546 6734 raft_consensus.cc:385] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.027590 6734 raft_consensus.cc:740] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.027545 6731 raft_consensus.cc:359] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.027663 6734 consensus_queue.cc:260] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.027791 6734 ts_tablet_manager.cc:1434] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.027911 6734 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.027666 6731 raft_consensus.cc:385] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.028184 6731 raft_consensus.cc:740] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.028245 6731 consensus_queue.cc:260] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.028343 6731 ts_tablet_manager.cc:1434] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.028383 6734 tablet_bootstrap.cc:654] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.028412 6731 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.028868 6731 tablet_bootstrap.cc:654] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.029197 6734 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.027851 6730 tablet_bootstrap.cc:654] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.029256 6734 ts_tablet_manager.cc:1403] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.029419 6734 raft_consensus.cc:359] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.029495 6734 raft_consensus.cc:385] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.029507 6731 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.029567 6731 ts_tablet_manager.cc:1403] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.029521 6734 raft_consensus.cc:740] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.029649 6734 consensus_queue.cc:260] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.029722 6731 raft_consensus.cc:359] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.029793 6731 raft_consensus.cc:385] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.029867 6731 raft_consensus.cc:740] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.029911 6731 consensus_queue.cc:260] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.030006 6731 ts_tablet_manager.cc:1434] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.030077 6731 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.030488 6730 tablet_bootstrap.cc:492] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.030525 6731 tablet_bootstrap.cc:654] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.030546 6730 ts_tablet_manager.cc:1403] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 09:10:36.030718 6730 raft_consensus.cc:359] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.030792 6730 raft_consensus.cc:385] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.030819 6730 raft_consensus.cc:740] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.030869 6730 consensus_queue.cc:260] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.029752 6734 ts_tablet_manager.cc:1434] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.030965 6734 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.031186 6730 ts_tablet_manager.cc:1434] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.031258 6730 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.031380 6731 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.031432 6731 ts_tablet_manager.cc:1403] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.031430 6734 tablet_bootstrap.cc:654] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.031610 6731 raft_consensus.cc:359] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.031677 6731 raft_consensus.cc:385] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.031703 6731 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.031725 6730 tablet_bootstrap.cc:654] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.031751 6731 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.031884 6731 ts_tablet_manager.cc:1434] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.031956 6731 tablet_bootstrap.cc:492] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.032346 6730 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.032393 6730 ts_tablet_manager.cc:1403] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.032406 6731 tablet_bootstrap.cc:654] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.032536 6730 raft_consensus.cc:359] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.032603 6730 raft_consensus.cc:385] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.032618 6730 raft_consensus.cc:740] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.032660 6730 consensus_queue.cc:260] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.032757 6730 ts_tablet_manager.cc:1434] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:36.032826 6730 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.033254 6730 tablet_bootstrap.cc:654] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.033821 6734 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.033828 6730 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.033878 6734 ts_tablet_manager.cc:1403] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251026 09:10:36.033895 6730 ts_tablet_manager.cc:1403] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.034057 6734 raft_consensus.cc:359] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.034060 6730 raft_consensus.cc:359] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.034132 6734 raft_consensus.cc:385] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.034215 6730 raft_consensus.cc:385] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.034224 6734 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.034256 6730 raft_consensus.cc:740] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.034287 6734 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.034323 6730 consensus_queue.cc:260] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.034425 6730 ts_tablet_manager.cc:1434] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.034487 6730 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70: Bootstrap starting.
I20251026 09:10:36.034610 6731 tablet_bootstrap.cc:492] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.034657 6731 ts_tablet_manager.cc:1403] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20251026 09:10:36.034677 6734 ts_tablet_manager.cc:1434] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.034744 6734 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.034826 6731 raft_consensus.cc:359] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.034885 6731 raft_consensus.cc:385] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.034905 6731 raft_consensus.cc:740] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.034950 6730 tablet_bootstrap.cc:654] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.034970 6731 consensus_queue.cc:260] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.035077 6731 ts_tablet_manager.cc:1434] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.035133 6731 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.035151 6734 tablet_bootstrap.cc:654] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.035632 6731 tablet_bootstrap.cc:654] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.036208 6734 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.036208 6730 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70: No bootstrap required, opened a new log
I20251026 09:10:36.036295 6734 ts_tablet_manager.cc:1403] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:10:36.036298 6730 ts_tablet_manager.cc:1403] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:10:36.036337 6731 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.036419 6731 ts_tablet_manager.cc:1403] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.036477 6734 raft_consensus.cc:359] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.036530 6730 raft_consensus.cc:359] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.036567 6734 raft_consensus.cc:385] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.036602 6734 raft_consensus.cc:740] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.036576 6730 raft_consensus.cc:385] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.036674 6730 raft_consensus.cc:740] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.036571 6731 raft_consensus.cc:359] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.036685 6734 consensus_queue.cc:260] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.036718 6730 consensus_queue.cc:260] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.036801 6734 ts_tablet_manager.cc:1434] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:10:36.036872 6730 ts_tablet_manager.cc:1434] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.036717 6731 raft_consensus.cc:385] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.036953 6731 raft_consensus.cc:740] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.037029 6731 consensus_queue.cc:260] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.037127 6731 ts_tablet_manager.cc:1434] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:10:36.037220 6731 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: Bootstrap starting.
I20251026 09:10:36.036881 6734 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:10:36.037714 6731 tablet_bootstrap.cc:654] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.038028 6734 tablet_bootstrap.cc:654] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Neither blocks nor log segments found. Creating new log.
I20251026 09:10:36.038405 6731 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: No bootstrap required, opened a new log
I20251026 09:10:36.038450 6731 ts_tablet_manager.cc:1403] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.038600 6731 raft_consensus.cc:359] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.038606 6734 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: No bootstrap required, opened a new log
I20251026 09:10:36.038669 6731 raft_consensus.cc:385] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.038691 6734 ts_tablet_manager.cc:1403] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20251026 09:10:36.038697 6731 raft_consensus.cc:740] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.038800 6731 consensus_queue.cc:260] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.038866 6734 raft_consensus.cc:359] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.038928 6734 raft_consensus.cc:385] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:10:36.038946 6734 raft_consensus.cc:740] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: FOLLOWER
I20251026 09:10:36.038964 6731 ts_tablet_manager.cc:1434] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:10:36.039088 6734 consensus_queue.cc:260] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.039243 6734 ts_tablet_manager.cc:1434] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.043939 6739 raft_consensus.cc:493] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.044071 6739 raft_consensus.cc:515] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.044339 6739 leader_election.cc:290] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:36.046283 6737 raft_consensus.cc:493] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.046406 6737 raft_consensus.cc:515] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.046617 6737 leader_election.cc:290] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.047919 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad759f01ba6c4154968854b4b9378651" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c" is_pre_election: true
I20251026 09:10:36.048051 6533 raft_consensus.cc:2468] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 0.
I20251026 09:10:36.048194 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad759f01ba6c4154968854b4b9378651" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.048255 6609 raft_consensus.cc:2468] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 0.
I20251026 09:10:36.048420 6650 leader_election.cc:304] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.048554 6739 raft_consensus.cc:2804] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.048604 6739 raft_consensus.cc:493] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.048624 6739 raft_consensus.cc:3060] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.049433 6739 raft_consensus.cc:515] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.049597 6739 leader_election.cc:290] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:36.049757 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad759f01ba6c4154968854b4b9378651" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.049825 6609 raft_consensus.cc:3060] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.049784 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad759f01ba6c4154968854b4b9378651" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:36.049867 6533 raft_consensus.cc:3060] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.050134 6608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.050199 6608 raft_consensus.cc:2468] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.050381 6499 leader_election.cc:304] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.050462 6737 raft_consensus.cc:2804] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.050511 6737 raft_consensus.cc:493] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.050539 6737 raft_consensus.cc:3060] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.050854 6533 raft_consensus.cc:2468] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 1.
I20251026 09:10:36.051216 6647 leader_election.cc:304] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.051218 6609 raft_consensus.cc:2468] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 1.
I20251026 09:10:36.051316 6737 raft_consensus.cc:515] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.051542 6737 leader_election.cc:290] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.051544 6739 raft_consensus.cc:2804] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.051708 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.051744 6739 raft_consensus.cc:697] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:36.051774 6609 raft_consensus.cc:3060] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.051806 6739 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.052484 6609 raft_consensus.cc:2468] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.052452 6430 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 1 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } }
I20251026 09:10:36.052645 6499 leader_election.cc:304] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.052683 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" is_pre_election: true
I20251026 09:10:36.052721 6737 raft_consensus.cc:2804] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.052754 6684 raft_consensus.cc:2468] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.052877 6737 raft_consensus.cc:697] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:36.052973 6737 consensus_queue.cc:237] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.052728 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.053226 6683 raft_consensus.cc:3060] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.053545 6430 catalog_manager.cc:5649] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 1 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:36.053908 6683 raft_consensus.cc:2468] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.057164 6733 raft_consensus.cc:493] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.057253 6733 raft_consensus.cc:515] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.057417 6733 leader_election.cc:290] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.060287 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c" is_pre_election: true
I20251026 09:10:36.060300 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" is_pre_election: true
I20251026 09:10:36.060420 6533 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 0.
I20251026 09:10:36.060427 6683 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 0.
I20251026 09:10:36.060606 6572 leader_election.cc:304] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.060719 6733 raft_consensus.cc:2804] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.060766 6733 raft_consensus.cc:493] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.060793 6733 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.061486 6733 raft_consensus.cc:515] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.061640 6733 leader_election.cc:290] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 1 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.061858 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.061892 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:36.061926 6683 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.061978 6533 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.062582 6683 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 1.
I20251026 09:10:36.062687 6533 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 1.
I20251026 09:10:36.062790 6574 leader_election.cc:304] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.062906 6733 raft_consensus.cc:2804] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.063092 6733 raft_consensus.cc:697] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:36.063153 6733 consensus_queue.cc:237] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.063838 6430 catalog_manager.cc:5649] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:36.106033 6739 raft_consensus.cc:493] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.106163 6739 raft_consensus.cc:515] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.106365 6739 leader_election.cc:290] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:36.106660 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.106637 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c" is_pre_election: true
I20251026 09:10:36.106781 6533 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 0.
I20251026 09:10:36.106779 6609 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 0.
I20251026 09:10:36.106969 6647 leader_election.cc:304] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.107105 6739 raft_consensus.cc:2804] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.107172 6739 raft_consensus.cc:493] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.107203 6739 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.107961 6739 raft_consensus.cc:515] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.108150 6739 leader_election.cc:290] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:36.108367 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:36.108379 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.108453 6533 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.108464 6609 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.109071 6609 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 1.
I20251026 09:10:36.109103 6533 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 1.
I20251026 09:10:36.109275 6650 leader_election.cc:304] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.109372 6739 raft_consensus.cc:2804] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.109438 6739 raft_consensus.cc:697] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:36.109506 6739 consensus_queue.cc:237] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.110332 6430 catalog_manager.cc:5649] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 1 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } }
I20251026 09:10:36.123214 6737 raft_consensus.cc:493] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.123382 6737 raft_consensus.cc:515] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.123575 6737 leader_election.cc:290] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.123827 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" is_pre_election: true
I20251026 09:10:36.123910 6683 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.123845 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.123955 6609 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.124058 6498 leader_election.cc:304] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.124184 6737 raft_consensus.cc:2804] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.124230 6737 raft_consensus.cc:493] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.124246 6737 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.124979 6737 raft_consensus.cc:515] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.125131 6737 leader_election.cc:290] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.125295 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.125345 6609 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.125330 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.125413 6683 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.126013 6609 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.126024 6683 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.126211 6499 leader_election.cc:304] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.126323 6737 raft_consensus.cc:2804] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.126392 6737 raft_consensus.cc:697] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:36.126452 6737 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.127079 6430 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 1 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:36.157212 6733 raft_consensus.cc:493] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.157342 6733 raft_consensus.cc:515] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.157558 6733 leader_election.cc:290] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.157837 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c" is_pre_election: true
I20251026 09:10:36.157859 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" is_pre_election: true
I20251026 09:10:36.157948 6533 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 0.
I20251026 09:10:36.157953 6683 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 0.
I20251026 09:10:36.158103 6572 leader_election.cc:304] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.158202 6733 raft_consensus.cc:2804] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.158243 6733 raft_consensus.cc:493] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.158268 6733 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.158957 6733 raft_consensus.cc:515] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.159101 6733 leader_election.cc:290] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 1 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.159374 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:36.159376 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.159473 6533 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.159489 6683 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.160068 6533 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 1.
I20251026 09:10:36.160077 6683 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 1.
I20251026 09:10:36.160275 6574 leader_election.cc:304] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.160373 6733 raft_consensus.cc:2804] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.160435 6733 raft_consensus.cc:697] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:36.160494 6733 consensus_queue.cc:237] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.161222 6430 catalog_manager.cc:5649] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:36.174938 6737 raft_consensus.cc:493] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.175071 6737 raft_consensus.cc:515] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.175261 6737 leader_election.cc:290] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.175525 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" is_pre_election: true
I20251026 09:10:36.175606 6683 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.175532 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.175653 6609 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.175760 6498 leader_election.cc:304] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.175904 6737 raft_consensus.cc:2804] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.175948 6737 raft_consensus.cc:493] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.175971 6737 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.176607 6739 raft_consensus.cc:493] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.176607 6737 raft_consensus.cc:515] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.176693 6739 raft_consensus.cc:515] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.176788 6739 leader_election.cc:290] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:36.176924 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.176952 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c" is_pre_election: true
I20251026 09:10:36.176978 6683 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.176999 6533 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 0.
I20251026 09:10:36.177017 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.177078 6779 raft_consensus.cc:493] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.177090 6609 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 0.
I20251026 09:10:36.177120 6779 raft_consensus.cc:515] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.177220 6779 leader_election.cc:290] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.177248 6608 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.177305 6608 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.177394 6737 leader_election.cc:290] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.177441 6647 leader_election.cc:304] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.177381 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.177500 6609 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.177516 6739 raft_consensus.cc:2804] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.177554 6739 raft_consensus.cc:493] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.177579 6739 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.177610 6499 leader_election.cc:304] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.177685 6737 raft_consensus.cc:2804] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.177709 6737 raft_consensus.cc:493] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.177728 6737 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.178031 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" is_pre_election: true
I20251026 09:10:36.178103 6684 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.178159 6608 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.178292 6499 leader_election.cc:304] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.178361 6779 raft_consensus.cc:2804] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.178339 6739 raft_consensus.cc:515] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.178339 6737 raft_consensus.cc:515] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.178504 6737 leader_election.cc:290] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.178535 6779 raft_consensus.cc:697] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:36.178598 6779 consensus_queue.cc:237] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.178658 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:36.178779 6533 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.178928 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.178995 6609 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.178709 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.179029 6684 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.179512 6533 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 1.
I20251026 09:10:36.179569 6684 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.178508 6739 leader_election.cc:290] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:36.179617 6609 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 1.
I20251026 09:10:36.179688 6647 leader_election.cc:304] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.179708 6498 leader_election.cc:304] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.179750 6739 raft_consensus.cc:2804] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.179785 6779 raft_consensus.cc:2804] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.179832 6779 raft_consensus.cc:697] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:36.179883 6779 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.178673 6683 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.180251 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.179787 6739 raft_consensus.cc:697] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:36.180335 6609 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.180370 6739 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.180361 6430 catalog_manager.cc:5649] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 1 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:36.180900 6609 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.181202 6430 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 1 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } }
I20251026 09:10:36.181433 6429 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 1 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:36.294214 6739 raft_consensus.cc:493] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.294365 6739 raft_consensus.cc:515] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.294544 6739 leader_election.cc:290] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:36.294822 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c" is_pre_election: true
I20251026 09:10:36.294853 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.295046 6533 raft_consensus.cc:2468] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 0.
I20251026 09:10:36.295063 6609 raft_consensus.cc:2468] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 0.
I20251026 09:10:36.295101 6788 raft_consensus.cc:493] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.295203 6788 raft_consensus.cc:515] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.295236 6647 leader_election.cc:304] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.295320 6739 raft_consensus.cc:2804] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.295367 6788 leader_election.cc:290] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.295385 6739 raft_consensus.cc:493] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.295410 6739 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.295493 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22a8d92b27b14f89a4728a8f50db932b" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.295503 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22a8d92b27b14f89a4728a8f50db932b" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" is_pre_election: true
I20251026 09:10:36.295578 6609 raft_consensus.cc:2468] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.295588 6683 raft_consensus.cc:2468] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.295722 6499 leader_election.cc:304] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.295814 6788 raft_consensus.cc:2804] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.295874 6788 raft_consensus.cc:493] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.295894 6788 raft_consensus.cc:3060] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.296315 6739 raft_consensus.cc:515] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.296454 6739 leader_election.cc:290] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:36.296648 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.296663 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:36.296670 6788 raft_consensus.cc:515] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.296729 6609 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.296761 6533 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.296850 6788 leader_election.cc:290] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.297005 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22a8d92b27b14f89a4728a8f50db932b" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.297061 6683 raft_consensus.cc:3060] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.297425 6609 raft_consensus.cc:2468] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 1.
I20251026 09:10:36.297520 6533 raft_consensus.cc:2468] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 1.
I20251026 09:10:36.297582 6650 leader_election.cc:304] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.297634 6683 raft_consensus.cc:2468] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.297683 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22a8d92b27b14f89a4728a8f50db932b" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.297688 6739 raft_consensus.cc:2804] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.297739 6609 raft_consensus.cc:3060] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.297763 6739 raft_consensus.cc:697] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:36.297770 6498 leader_election.cc:304] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.297822 6739 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.297870 6788 raft_consensus.cc:2804] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.297925 6788 raft_consensus.cc:697] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:36.297971 6788 consensus_queue.cc:237] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.298422 6609 raft_consensus.cc:2468] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.298692 6430 catalog_manager.cc:5649] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 1 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:36.298692 6429 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 1 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } }
I20251026 09:10:36.307442 6788 raft_consensus.cc:493] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:10:36.307556 6788 raft_consensus.cc:515] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.307709 6788 leader_election.cc:290] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.307967 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" is_pre_election: true
I20251026 09:10:36.308053 6683 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.307971 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361" is_pre_election: true
I20251026 09:10:36.308152 6609 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 0.
I20251026 09:10:36.308202 6498 leader_election.cc:304] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.308286 6788 raft_consensus.cc:2804] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:10:36.308320 6788 raft_consensus.cc:493] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:10:36.308347 6788 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.309016 6788 raft_consensus.cc:515] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.309158 6788 leader_election.cc:290] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.309307 6683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.309366 6683 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.309350 6609 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.309453 6609 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:10:36.309979 6683 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.310110 6609 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 1.
I20251026 09:10:36.310175 6498 leader_election.cc:304] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:36.310273 6788 raft_consensus.cc:2804] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:10:36.310323 6788 raft_consensus.cc:697] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:36.310376 6788 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.311024 6429 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 1 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:36.344424 6683 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.344810 6683 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.345101 6683 raft_consensus.cc:1275] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.345055 6532 raft_consensus.cc:1275] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.345201 6607 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.345070 6733 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.345307 6533 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.345149 6608 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.345425 6733 consensus_queue.cc:1048] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.345046 6609 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.345700 6779 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.345837 6533 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.345847 6532 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.346005 6783 consensus_queue.cc:1048] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.346119 6783 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.346524 6609 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.346540 6608 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.346014 6739 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 09:10:36.346885 6737 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.346930 6739 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.346963 6608 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.346132 6777 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 09:10:36.346964 6737 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.347182 6779 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.348387 6810 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.348621 6810 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.348701 6609 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.350208 6779 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.350346 6681 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.350484 6680 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.350595 6678 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.350586 6679 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.350757 6737 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.351292 6788 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.351322 6737 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.351434 6788 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.351809 6603 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.353471 6603 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.353937 6777 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.354614 6837 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251026 09:10:36.362629 6532 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.363430 6532 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:10:36.363976 6777 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.365116 6677 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.367625 6788 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.365373 6840 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251026 09:10:36.368585 6815 mvcc.cc:204] Tried to move back new op lower bound from 7214980449665273856 to 7214980448468881408. Current Snapshot: MvccSnapshot[applied={T|T < 7214980449665273856}]
I20251026 09:10:36.368672 6601 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:10:36.371969 6788 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:10:36.375855 6806 mvcc.cc:204] Tried to move back new op lower bound from 7214980449675116544 to 7214980449476943872. Current Snapshot: MvccSnapshot[applied={T|T < 7214980449675116544}]
I20251026 09:10:36.386298 6815 mvcc.cc:204] Tried to move back new op lower bound from 7214980449767759872 to 7214980448774651904. Current Snapshot: MvccSnapshot[applied={T|T < 7214980449767759872 or (T in {7214980449767759872})}]
I20251026 09:10:36.541918 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:36.543625 6892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:36.543983 3128 server_base.cc:1047] running on GCE node
W20251026 09:10:36.545397 6893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:36.545416 6895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:36.545818 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:36.545866 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:36.545887 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469836545887 us; error 0 us; skew 500 ppm
I20251026 09:10:36.546723 3128 webserver.cc:492] Webserver started at http://127.3.14.4:41597/ using document root <none> and password file <none>
I20251026 09:10:36.546818 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:36.546880 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:36.546936 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:36.547204 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-3-root/instance:
uuid: "26c251f040b9472a92191f35891e8eed"
format_stamp: "Formatted at 2025-10-26 09:10:36 on dist-test-slave-w8k0"
I20251026 09:10:36.548210 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:10:36.548674 6900 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.548861 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.548907 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-3-root
uuid: "26c251f040b9472a92191f35891e8eed"
format_stamp: "Formatted at 2025-10-26 09:10:36 on dist-test-slave-w8k0"
I20251026 09:10:36.548957 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:36.558899 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:36.559140 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:36.559278 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:36.559521 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:36.559554 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.559590 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:36.559620 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.563280 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.4:39841
I20251026 09:10:36.563424 6963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.4:39841 every 8 connection(s)
I20251026 09:10:36.563825 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:36.565472 6969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:36.565564 6971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:36.565730 6968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:36.565752 3128 server_base.cc:1047] running on GCE node
I20251026 09:10:36.565968 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:36.566004 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:36.566030 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469836566029 us; error 0 us; skew 500 ppm
I20251026 09:10:36.567063 3128 webserver.cc:492] Webserver started at http://127.3.14.5:38349/ using document root <none> and password file <none>
I20251026 09:10:36.567171 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:36.567230 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:36.567284 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:36.567463 6964 heartbeater.cc:344] Connected to a master server at 127.3.14.62:38937
I20251026 09:10:36.567523 6964 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:36.567662 6964 heartbeater.cc:507] Master 127.3.14.62:38937 requested a full tablet report, sending...
I20251026 09:10:36.567684 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-4-root/instance:
uuid: "90365b2687854499bddaa63c08c0964f"
format_stamp: "Formatted at 2025-10-26 09:10:36 on dist-test-slave-w8k0"
I20251026 09:10:36.567906 6430 ts_manager.cc:194] Registered new tserver with Master: 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841)
I20251026 09:10:36.568363 6430 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43332
I20251026 09:10:36.568734 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.569275 6976 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.569415 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:36.569465 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-4-root
uuid: "90365b2687854499bddaa63c08c0964f"
format_stamp: "Formatted at 2025-10-26 09:10:36 on dist-test-slave-w8k0"
I20251026 09:10:36.569512 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:36.581194 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:36.581501 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:36.581666 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:36.581907 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:36.581940 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.581965 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:36.581992 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.585786 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.5:40007
I20251026 09:10:36.585896 7039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.5:40007 every 8 connection(s)
I20251026 09:10:36.586277 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:10:36.587659 7043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:10:36.587792 7046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:36.587846 3128 server_base.cc:1047] running on GCE node
W20251026 09:10:36.587997 7044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:10:36.588200 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:10:36.588258 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:10:36.588284 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469836588284 us; error 0 us; skew 500 ppm
I20251026 09:10:36.589212 3128 webserver.cc:492] Webserver started at http://127.3.14.6:42727/ using document root <none> and password file <none>
I20251026 09:10:36.589318 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:10:36.589376 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:10:36.589434 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:10:36.589807 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-5-root/instance:
uuid: "485b30c027fe4845b029de91594146fd"
format_stamp: "Formatted at 2025-10-26 09:10:36 on dist-test-slave-w8k0"
I20251026 09:10:36.589990 7040 heartbeater.cc:344] Connected to a master server at 127.3.14.62:38937
I20251026 09:10:36.590061 7040 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:36.590171 7040 heartbeater.cc:507] Master 127.3.14.62:38937 requested a full tablet report, sending...
I20251026 09:10:36.590416 6430 ts_manager.cc:194] Registered new tserver with Master: 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007)
I20251026 09:10:36.590756 6430 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43342
I20251026 09:10:36.591075 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:10:36.591682 7051 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.591820 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.591862 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-5-root
uuid: "485b30c027fe4845b029de91594146fd"
format_stamp: "Formatted at 2025-10-26 09:10:36 on dist-test-slave-w8k0"
I20251026 09:10:36.591923 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:10:36.601985 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:10:36.602314 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:10:36.602635 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:10:36.602849 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:10:36.602887 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.602910 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:10:36.602938 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:36.606379 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.6:35539
I20251026 09:10:36.606557 7114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.6:35539 every 8 connection(s)
I20251026 09:10:36.610121 7115 heartbeater.cc:344] Connected to a master server at 127.3.14.62:38937
I20251026 09:10:36.610199 7115 heartbeater.cc:461] Registering TS with master...
I20251026 09:10:36.610318 7115 heartbeater.cc:507] Master 127.3.14.62:38937 requested a full tablet report, sending...
I20251026 09:10:36.610564 6430 ts_manager.cc:194] Registered new tserver with Master: 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:10:36.610924 6430 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43352
I20251026 09:10:36.825976 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:36.829738 6533 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:36.829809 6532 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:36.830013 6532 raft_consensus.cc:606] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:10:36.830478 6533 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:36.830551 6681 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20251026 09:10:36.830621 6604 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20251026 09:10:36.830552 6533 raft_consensus.cc:606] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:10:36.830799 6879 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251026 09:10:36.830924 6878 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251026 09:10:36.831548 6873 raft_consensus.cc:993] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:36.831604 6873 raft_consensus.cc:1081] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:36.831658 6873 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index -1 to 36, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:36.831661 6604 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index -1 to 36, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:36.833043 6430 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index -1 to 36, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New cstate: current_term: 1 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:36.833683 6604 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47314
I20251026 09:10:36.833751 6604 raft_consensus.cc:493] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:36.833781 6604 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:36.834055 6681 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index -1 to 36, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:36.834591 6604 raft_consensus.cc:515] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:36.834749 6604 leader_election.cc:290] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.834884 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:36.834954 6533 raft_consensus.cc:3055] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:36.834977 6533 raft_consensus.cc:740] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:36.835029 6533 consensus_queue.cc:260] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:36.835114 6533 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:36.835654 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:36.835721 6684 raft_consensus.cc:606] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:36.835898 6533 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:36.836058 6572 leader_election.cc:304] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.836233 6783 raft_consensus.cc:2804] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:36.836301 6783 raft_consensus.cc:697] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:36.836364 6783 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:36.836532 6533 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:36.836539 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.836607 6533 raft_consensus.cc:606] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:10:36.836614 6681 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:36.837280 6681 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:36.837365 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251026 09:10:36.837589 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:36.837575 6428 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 1 to 2, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 09:10:36.840454 6478 auto_rebalancer.cc:253] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20251026 09:10:36.840981 6604 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47308
I20251026 09:10:36.841073 6604 raft_consensus.cc:606] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Received request to transfer leadership
W20251026 09:10:36.841641 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 90072194423a4007b85cbc3f31889400, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:36.841687 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 90072194423a4007b85cbc3f31889400, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.842041 6604 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47308
I20251026 09:10:36.842108 6604 raft_consensus.cc:606] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Received request to transfer leadership
I20251026 09:10:36.842445 6604 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47308
I20251026 09:10:36.842499 6604 raft_consensus.cc:606] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Received request to transfer leadership
W20251026 09:10:36.842629 6478 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 75fb5eb7df3b4c87bfdaa0b3f0efe601 already in progress
W20251026 09:10:36.842665 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 75fb5eb7df3b4c87bfdaa0b3f0efe601 already in progress
W20251026 09:10:36.843151 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e65f6e7109bc4eda8d83d836786c4d16, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:36.843199 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet e65f6e7109bc4eda8d83d836786c4d16, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:36.843634 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 21bd52ca927842ccab72fc70494128d0, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:36.843670 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 21bd52ca927842ccab72fc70494128d0, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:36.844024 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e002d8e9fb004210be27d87c5a23f9c3, TS cfcab08da3db40e1974902b787ef6361 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:36.844059 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet e002d8e9fb004210be27d87c5a23f9c3, TS cfcab08da3db40e1974902b787ef6361 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:36.844370 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ac1d836cf54d42c39eb2d8279fd17f90, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:36.844406 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ac1d836cf54d42c39eb2d8279fd17f90, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.982309 6887 raft_consensus.cc:993] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:36.982404 6887 raft_consensus.cc:1081] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:36.982607 6533 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44252
I20251026 09:10:36.982875 6533 raft_consensus.cc:493] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:36.982908 6533 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:36.983681 6533 raft_consensus.cc:515] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.983842 6533 leader_election.cc:290] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:36.984011 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:36.984021 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:36.984095 6604 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:36.984108 6681 raft_consensus.cc:3055] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:36.984133 6681 raft_consensus.cc:740] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:36.984198 6681 consensus_queue.cc:260] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.984292 6681 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:36.984810 6604 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:36.984846 6681 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:36.985020 6499 leader_election.cc:304] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:36.985121 6879 raft_consensus.cc:2804] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:36.985188 6879 raft_consensus.cc:697] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:36.985247 6879 consensus_queue.cc:237] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:36.985884 6427 catalog_manager.cc:5649] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 1 to 2, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 2 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:37.036931 6879 raft_consensus.cc:993] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:37.037016 6879 raft_consensus.cc:1081] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:37.037220 6604 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47314
I20251026 09:10:37.037286 6604 raft_consensus.cc:493] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:37.037307 6604 raft_consensus.cc:3060] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:37.037951 6604 raft_consensus.cc:515] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:37.038082 6604 leader_election.cc:290] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:37.038249 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:37.038319 6533 raft_consensus.cc:3055] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:37.038339 6533 raft_consensus.cc:740] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:37.038287 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:37.038388 6681 raft_consensus.cc:3060] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:37.038388 6533 consensus_queue.cc:260] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 35, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:37.038507 6533 raft_consensus.cc:3060] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:37.039007 6681 raft_consensus.cc:2468] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:37.039067 6533 raft_consensus.cc:2468] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:37.039214 6574 leader_election.cc:304] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:37.039314 6783 raft_consensus.cc:2804] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:37.039417 6783 raft_consensus.cc:697] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:37.039459 6783 consensus_queue.cc:237] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 35, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:37.039958 6427 catalog_manager.cc:5649] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 1 to 2, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:37.043071 6860 raft_consensus.cc:993] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:37.043293 6860 raft_consensus.cc:1081] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:37.043601 6604 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47314
I20251026 09:10:37.043704 6604 raft_consensus.cc:493] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:37.043735 6604 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:37.044397 6604 raft_consensus.cc:515] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:37.044725 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:37.044739 6604 leader_election.cc:290] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:37.044816 6533 raft_consensus.cc:3055] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:37.044840 6533 raft_consensus.cc:740] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:37.044728 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:37.044916 6681 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:37.044924 6533 consensus_queue.cc:260] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:37.045063 6533 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:37.045614 6681 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:37.045651 6533 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:37.045775 6574 leader_election.cc:304] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:37.045879 6783 raft_consensus.cc:2804] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:37.045943 6783 raft_consensus.cc:697] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:37.046000 6783 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:37.046523 6427 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 1 to 2, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:37.364005 6681 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:37.364392 6860 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:37.365880 6604 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:37.366210 7129 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:37.427903 6681 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:37.428246 6783 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
W20251026 09:10:37.429095 6575 consensus_peers.cc:597] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 -> Peer 485b30c027fe4845b029de91594146fd (127.3.14.6:35539): Couldn't send request to peer 485b30c027fe4845b029de91594146fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75fb5eb7df3b4c87bfdaa0b3f0efe601. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:37.429189 6891 raft_consensus.cc:993] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:37.429395 6891 raft_consensus.cc:1081] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:37.429721 6681 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33638
I20251026 09:10:37.429804 6681 raft_consensus.cc:493] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:37.429838 6681 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:37.430045 6533 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:37.430389 7134 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:37.430775 6681 raft_consensus.cc:515] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:37.430927 6681 leader_election.cc:290] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:37.431403 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:37.431478 6604 raft_consensus.cc:3055] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:10:37.431507 6604 raft_consensus.cc:740] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:37.431562 6604 consensus_queue.cc:260] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:37.431669 6604 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:37.431973 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:37.432078 6533 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:37.432601 6604 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:37.432816 6533 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:37.432874 6650 leader_election.cc:304] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:37.432964 6887 raft_consensus.cc:2804] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:10:37.433007 6887 raft_consensus.cc:697] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:37.433063 6887 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.37, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:37.433331 6681 raft_consensus.cc:1240] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Rejecting Update request from peer cfcab08da3db40e1974902b787ef6361 for earlier term 2. Current term is 3. Ops: []
I20251026 09:10:37.435572 6427 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 2 to 3, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:37.439085 6681 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20251026 09:10:37.439364 7136 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251026 09:10:37.440589 6533 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20251026 09:10:37.440794 6783 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251026 09:10:37.567997 6681 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:37.568243 6783 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:37.568948 6964 heartbeater.cc:499] Master 127.3.14.62:38937 was elected leader, sending a full tablet report...
I20251026 09:10:37.569657 6533 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:37.570039 7134 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:37.591425 7040 heartbeater.cc:499] Master 127.3.14.62:38937 was elected leader, sending a full tablet report...
I20251026 09:10:37.611743 7115 heartbeater.cc:499] Master 127.3.14.62:38937 was elected leader, sending a full tablet report...
I20251026 09:10:37.844298 6533 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:37.844667 6885 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
W20251026 09:10:37.845494 6478 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20251026 09:10:37.846148 6604 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
W20251026 09:10:37.846204 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601, TS cfcab08da3db40e1974902b787ef6361 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
W20251026 09:10:37.846259 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601, TS cfcab08da3db40e1974902b787ef6361 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:37.846340 7139 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
W20251026 09:10:37.847056 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e002d8e9fb004210be27d87c5a23f9c3, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.847100 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet e002d8e9fb004210be27d87c5a23f9c3, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.847595 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ff5ceda9af754029a9b5ac636a17ea70, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.847667 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ff5ceda9af754029a9b5ac636a17ea70, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.848161 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4bb5819b36e244d4b0ab2c0e110e51e9, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.848207 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 4bb5819b36e244d4b0ab2c0e110e51e9, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.848677 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 90072194423a4007b85cbc3f31889400, TS cfcab08da3db40e1974902b787ef6361 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.848726 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 90072194423a4007b85cbc3f31889400, TS cfcab08da3db40e1974902b787ef6361 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.849084 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ac1d836cf54d42c39eb2d8279fd17f90, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.849130 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ac1d836cf54d42c39eb2d8279fd17f90, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:37.849859 6650 consensus_peers.cc:597] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 485b30c027fe4845b029de91594146fd (127.3.14.6:35539): Couldn't send request to peer 485b30c027fe4845b029de91594146fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75fb5eb7df3b4c87bfdaa0b3f0efe601. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:38.319918 7142 ts_tablet_manager.cc:933] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: Initiating tablet copy from peer 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:38.320138 7142 tablet_copy_client.cc:323] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:39689
I20251026 09:10:38.322980 6694 tablet_copy_service.cc:140] P 6d20fbac1f1243a0b383c9383a1d0a70: Received BeginTabletCopySession request for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 from peer 485b30c027fe4845b029de91594146fd ({username='slave'} at 127.0.0.1:33682)
I20251026 09:10:38.323091 6694 tablet_copy_service.cc:161] P 6d20fbac1f1243a0b383c9383a1d0a70: Beginning new tablet copy session on tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 from peer 485b30c027fe4845b029de91594146fd at {username='slave'} at 127.0.0.1:33682: session id = 485b30c027fe4845b029de91594146fd-75fb5eb7df3b4c87bfdaa0b3f0efe601
I20251026 09:10:38.323671 6694 tablet_copy_source_session.cc:215] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:38.324093 7142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:38.325281 7142 tablet_copy_client.cc:806] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:38.325412 7142 tablet_copy_client.cc:670] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:38.326122 7143 tablet_copy_client.cc:1025] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:38.837955 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:38.838625 6681 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:38.838719 6681 raft_consensus.cc:606] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:38.839355 6533 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:38.839422 6533 raft_consensus.cc:606] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:10:38.839689 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 09:10:38.839838 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 09:10:38.850314 6478 auto_rebalancer.cc:253] failed to send replica move request: Invalid argument: must modify a field when calling MODIFY_PEER
W20251026 09:10:38.850914 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601, TS cfcab08da3db40e1974902b787ef6361 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
W20251026 09:10:38.850970 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601, TS cfcab08da3db40e1974902b787ef6361 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
W20251026 09:10:38.851449 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f9723f81063e4f23bb2300f2c7bd50eb, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:38.851492 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet f9723f81063e4f23bb2300f2c7bd50eb, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:38.852003 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 21bd52ca927842ccab72fc70494128d0, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:38.852056 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 21bd52ca927842ccab72fc70494128d0, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:38.852492 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ac1d836cf54d42c39eb2d8279fd17f90, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:38.852530 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ac1d836cf54d42c39eb2d8279fd17f90, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:38.852979 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6459af4a4180422a8eccf12fc862499e, TS cfcab08da3db40e1974902b787ef6361 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:38.853018 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 6459af4a4180422a8eccf12fc862499e, TS cfcab08da3db40e1974902b787ef6361 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:38.853505 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 90072194423a4007b85cbc3f31889400, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:38.853559 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 90072194423a4007b85cbc3f31889400, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:39.012674 6860 raft_consensus.cc:993] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:39.012785 6860 raft_consensus.cc:1081] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:39.013036 6604 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47314
I20251026 09:10:39.013140 6604 raft_consensus.cc:493] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:39.013185 6604 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:39.013921 6604 raft_consensus.cc:515] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:39.014082 6604 leader_election.cc:290] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:39.014230 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:39.014294 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:39.014336 6533 raft_consensus.cc:3055] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:39.014354 6681 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:39.014367 6533 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:39.014434 6533 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 35, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:39.014531 6533 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:39.015014 6681 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:39.015152 6533 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:39.015208 6574 leader_election.cc:304] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:39.015296 7134 raft_consensus.cc:2804] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:39.015442 7134 raft_consensus.cc:697] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:39.015538 7134 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 35, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:39.016274 6428 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 1 to 2, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:39.120656 7139 raft_consensus.cc:993] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:39.120782 7139 raft_consensus.cc:1081] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:39.121034 6533 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44252
I20251026 09:10:39.121135 6533 raft_consensus.cc:493] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:39.121160 6533 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:39.121836 6533 raft_consensus.cc:515] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:39.121989 6533 leader_election.cc:290] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:39.122162 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:39.122247 6604 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:39.122177 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:39.122292 6681 raft_consensus.cc:3055] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:39.122332 6681 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:39.122404 6681 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:39.122500 6681 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:39.123006 6604 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:39.123076 6681 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:39.123222 6499 leader_election.cc:304] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:39.123353 6860 raft_consensus.cc:2804] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:39.123409 6860 raft_consensus.cc:697] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:39.123456 6860 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:39.124142 6428 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 1 to 2, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 2 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:39.464659 6681 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20251026 09:10:39.464994 7134 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251026 09:10:39.466401 6533 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20251026 09:10:39.466600 7134 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251026 09:10:39.572948 6681 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:39.573241 6860 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:39.574384 6604 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:39.574828 7161 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:39.854750 6681 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:39.855604 6533 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20251026 09:10:39.855772 6604 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20251026 09:10:39.855846 7166 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:39.855970 7164 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:39.856690 7165 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:39.856822 6604 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:39.858067 6428 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index -1 to 37, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
W20251026 09:10:39.858609 6478 auto_rebalancer.cc:253] failed to send replica move request: Invalid argument: Server with UUID 485b30c027fe4845b029de91594146fd is already a member of the config. RaftConfig: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:39.858644 6533 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
W20251026 09:10:39.859452 6478 auto_rebalancer.cc:663] Could not move replica: Illegal state: 75fb5eb7df3b4c87bfdaa0b3f0efe601: source replica 3ed603c643ac44ca9908b171687d7d5c does not have REPLACE attribute set
W20251026 09:10:39.859505 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Illegal state: 75fb5eb7df3b4c87bfdaa0b3f0efe601: source replica 3ed603c643ac44ca9908b171687d7d5c does not have REPLACE attribute set
W20251026 09:10:39.859562 6650 consensus_peers.cc:597] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841): Couldn't send request to peer 26c251f040b9472a92191f35891e8eed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad759f01ba6c4154968854b4b9378651. This is attempt 1: this message will repeat every 5th retry.
W20251026 09:10:39.860384 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6459af4a4180422a8eccf12fc862499e, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:39.860425 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 6459af4a4180422a8eccf12fc862499e, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:39.861133 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 22a8d92b27b14f89a4728a8f50db932b, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:39.861177 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 22a8d92b27b14f89a4728a8f50db932b, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 90365b2687854499bddaa63c08c0964f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:39.861932 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ac1d836cf54d42c39eb2d8279fd17f90, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:39.862004 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ac1d836cf54d42c39eb2d8279fd17f90, TS 3ed603c643ac44ca9908b171687d7d5c -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:39.862701 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f9723f81063e4f23bb2300f2c7bd50eb, TS cfcab08da3db40e1974902b787ef6361 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:39.862742 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet f9723f81063e4f23bb2300f2c7bd50eb, TS cfcab08da3db40e1974902b787ef6361 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:40.326923 7142 tablet_copy_client.cc:538] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:40.328079 7142 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: Bootstrap starting.
I20251026 09:10:40.334627 7142 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=147 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:40.334982 7142 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: Bootstrap complete.
I20251026 09:10:40.335143 7142 ts_tablet_manager.cc:1403] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251026 09:10:40.335484 7142 raft_consensus.cc:359] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:40.335616 7142 raft_consensus.cc:740] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 485b30c027fe4845b029de91594146fd, State: Initialized, Role: LEARNER
I20251026 09:10:40.335657 7142 consensus_queue.cc:260] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:40.335937 7142 ts_tablet_manager.cc:1434] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20251026 09:10:40.336236 6694 tablet_copy_service.cc:342] P 6d20fbac1f1243a0b383c9383a1d0a70: Request end of tablet copy session 485b30c027fe4845b029de91594146fd-75fb5eb7df3b4c87bfdaa0b3f0efe601 received from {username='slave'} at 127.0.0.1:33682
I20251026 09:10:40.336297 6694 tablet_copy_service.cc:434] P 6d20fbac1f1243a0b383c9383a1d0a70: ending tablet copy session 485b30c027fe4845b029de91594146fd-75fb5eb7df3b4c87bfdaa0b3f0efe601 on tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 with peer 485b30c027fe4845b029de91594146fd
I20251026 09:10:40.479139 7178 ts_tablet_manager.cc:933] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: Initiating tablet copy from peer 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:40.479413 7178 tablet_copy_client.cc:323] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:39689
I20251026 09:10:40.482631 6694 tablet_copy_service.cc:140] P 6d20fbac1f1243a0b383c9383a1d0a70: Received BeginTabletCopySession request for tablet ad759f01ba6c4154968854b4b9378651 from peer 26c251f040b9472a92191f35891e8eed ({username='slave'} at 127.0.0.1:44952)
I20251026 09:10:40.482722 6694 tablet_copy_service.cc:161] P 6d20fbac1f1243a0b383c9383a1d0a70: Beginning new tablet copy session on tablet ad759f01ba6c4154968854b4b9378651 from peer 26c251f040b9472a92191f35891e8eed at {username='slave'} at 127.0.0.1:44952: session id = 26c251f040b9472a92191f35891e8eed-ad759f01ba6c4154968854b4b9378651
I20251026 09:10:40.483201 6694 tablet_copy_source_session.cc:215] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:40.483635 7178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad759f01ba6c4154968854b4b9378651. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:40.485142 7178 tablet_copy_client.cc:806] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:40.485282 7178 tablet_copy_client.cc:670] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:40.486173 7179 tablet_copy_client.cc:1025] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:40.537547 7089 raft_consensus.cc:1217] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 LEARNER]: Deduplicated request from leader. Original: 2.37->[3.38-3.38] Dedup: 3.38->[]
I20251026 09:10:40.840029 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:40.840453 6533 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:40.840562 6533 raft_consensus.cc:606] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Received request to transfer leadership to TS 6d20fbac1f1243a0b383c9383a1d0a70
I20251026 09:10:40.840754 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 09:10:40.840845 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:40.887853 7161 raft_consensus.cc:993] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:40.887967 7161 raft_consensus.cc:1081] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:40.888130 6681 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33632
I20251026 09:10:40.888208 6681 raft_consensus.cc:493] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:40.888236 6681 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:40.888892 6681 raft_consensus.cc:515] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:40.889040 6681 leader_election.cc:290] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:40.889178 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:40.889251 6604 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:40.890013 6604 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:40.890125 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:40.890188 6650 leader_election.cc:304] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:40.890188 6533 raft_consensus.cc:3055] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:10:40.890275 6533 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:40.890312 7146 raft_consensus.cc:2804] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:10:40.890337 6533 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:40.890413 7146 raft_consensus.cc:697] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:40.890432 6533 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:40.890457 7146 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:40.891083 6427 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 2 to 3, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } }
I20251026 09:10:40.891170 6533 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:41.134732 7146 raft_consensus.cc:1064] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70: attempting to promote NON_VOTER 485b30c027fe4845b029de91594146fd to VOTER
I20251026 09:10:41.134985 7146 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 37, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:41.135512 7089 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 LEARNER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:41.135551 6604 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:41.135533 6533 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:41.135789 7191 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:41.135887 7146 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:41.135947 7189 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:41.136929 7146 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.39: config changed from index 36 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:41.137130 7089 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 36 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:41.138195 6604 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 36 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:41.138192 6533 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 36 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:41.138370 6427 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 36 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:41.140156 6681 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 3.39, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:41.140537 7089 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251026 09:10:41.140561 6533 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251026 09:10:41.140726 7191 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:41.140815 7189 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:41.141350 7183 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:41.141449 6533 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:41.142282 6416 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:41.142477 6430 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 39 to 40, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:41.142596 7089 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:41.143222 6596 tablet_service.cc:1552] Processing DeleteTablet for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 with delete_type TABLET_DATA_TOMBSTONED (TS cfcab08da3db40e1974902b787ef6361 not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:47308
I20251026 09:10:41.143576 7196 tablet_replica.cc:333] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:10:41.143656 7196 raft_consensus.cc:2243] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:41.143752 7196 raft_consensus.cc:2272] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:41.144081 7196 ts_tablet_manager.cc:1916] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:41.145383 7196 ts_tablet_manager.cc:1929] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.39
I20251026 09:10:41.145437 7196 log.cc:1199] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P cfcab08da3db40e1974902b787ef6361: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/75fb5eb7df3b4c87bfdaa0b3f0efe601
I20251026 09:10:41.145793 6417 catalog_manager.cc:4985] TS cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631): tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:41.377996 6533 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:41.378244 7191 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:41.379624 6604 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:41.379828 7191 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:42.486406 7178 tablet_copy_client.cc:538] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:42.487448 7178 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: Bootstrap starting.
I20251026 09:10:42.492071 7178 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: Bootstrap replayed 1/1 log segments. Stats: ops{read=37 overwritten=0 applied=37 ignored=0} inserts{seen=148 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:42.492337 7178 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: Bootstrap complete.
I20251026 09:10:42.492484 7178 ts_tablet_manager.cc:1403] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 09:10:42.492690 7178 raft_consensus.cc:359] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:42.492782 7178 raft_consensus.cc:740] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Initialized, Role: LEARNER
I20251026 09:10:42.492823 7178 consensus_queue.cc:260] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 37, Last appended: 1.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:42.493005 7178 ts_tablet_manager.cc:1434] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: Time spent starting tablet: real 0.000s user 0.001s sys 0.001s
I20251026 09:10:42.493211 6694 tablet_copy_service.cc:342] P 6d20fbac1f1243a0b383c9383a1d0a70: Request end of tablet copy session 26c251f040b9472a92191f35891e8eed-ad759f01ba6c4154968854b4b9378651 received from {username='slave'} at 127.0.0.1:44952
I20251026 09:10:42.493287 6694 tablet_copy_service.cc:434] P 6d20fbac1f1243a0b383c9383a1d0a70: ending tablet copy session 26c251f040b9472a92191f35891e8eed-ad759f01ba6c4154968854b4b9378651 on tablet ad759f01ba6c4154968854b4b9378651 with peer 26c251f040b9472a92191f35891e8eed
I20251026 09:10:42.676162 6938 raft_consensus.cc:1217] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.36->[1.37-1.37] Dedup: 1.37->[]
I20251026 09:10:42.840972 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:42.841502 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:42.841609 6684 raft_consensus.cc:606] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:42.848503 6604 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:10:42.848641 6604 raft_consensus.cc:606] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:42.849045 6604 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:10:42.849130 6604 raft_consensus.cc:606] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Received request to transfer leadership to TS 6d20fbac1f1243a0b383c9383a1d0a70
I20251026 09:10:42.849300 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20251026 09:10:42.849391 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:42.885782 7174 raft_consensus.cc:993] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:42.885874 7174 raft_consensus.cc:1081] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:42.886070 6681 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33638
I20251026 09:10:42.886174 6681 raft_consensus.cc:493] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:42.886219 6681 raft_consensus.cc:3060] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:42.887133 6681 raft_consensus.cc:515] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:42.887288 6681 leader_election.cc:290] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:42.887440 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:42.887457 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e002d8e9fb004210be27d87c5a23f9c3" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:42.887559 6533 raft_consensus.cc:3060] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:42.887576 6604 raft_consensus.cc:3055] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:10:42.887605 6604 raft_consensus.cc:740] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:42.887669 6604 consensus_queue.cc:260] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 36, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:42.887753 6604 raft_consensus.cc:3060] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:42.888362 6604 raft_consensus.cc:2468] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:42.888362 6533 raft_consensus.cc:2468] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:42.888566 6647 leader_election.cc:304] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:42.888666 7189 raft_consensus.cc:2804] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:10:42.888844 7189 raft_consensus.cc:697] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:42.888916 7189 consensus_queue.cc:237] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 36, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:42.889532 6428 catalog_manager.cc:5649] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 2 to 3, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } }
I20251026 09:10:42.990303 7174 raft_consensus.cc:993] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:42.990404 7174 raft_consensus.cc:1081] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:42.990587 6533 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44258
I20251026 09:10:42.990660 6533 raft_consensus.cc:493] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:42.990684 6533 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:42.991663 6533 raft_consensus.cc:515] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:42.991835 6533 leader_election.cc:290] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:42.991979 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:42.992020 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:42.992131 6684 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:42.993067 6684 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:42.992053 6604 raft_consensus.cc:3055] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:42.993216 6604 raft_consensus.cc:740] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:42.993243 6498 leader_election.cc:304] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:42.993284 6604 consensus_queue.cc:260] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:42.993343 7161 raft_consensus.cc:2804] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:42.993391 7161 raft_consensus.cc:697] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:42.993395 6604 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:42.993448 7161 consensus_queue.cc:237] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:42.994056 6428 catalog_manager.cc:5649] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 1 to 2, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 2 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:42.994277 6604 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:43.147768 7189 raft_consensus.cc:993] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:43.147894 7189 raft_consensus.cc:1081] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:43.148077 6533 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44252
I20251026 09:10:43.148173 6533 raft_consensus.cc:493] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:43.148209 6533 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:43.148990 6533 raft_consensus.cc:515] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:43.149161 6533 leader_election.cc:290] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:43.149297 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:43.149384 6604 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:43.149394 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:43.149569 6681 raft_consensus.cc:3055] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:43.149596 6681 raft_consensus.cc:740] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:43.149667 6681 consensus_queue.cc:260] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 35, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:43.149802 6681 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:43.150214 6604 raft_consensus.cc:2468] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:43.150381 6499 leader_election.cc:304] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:43.150413 6681 raft_consensus.cc:2468] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:43.150511 7214 raft_consensus.cc:2804] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:43.150571 7214 raft_consensus.cc:697] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:43.150631 7214 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 35, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:43.151257 6428 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 1 to 2, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 2 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:43.264355 6533 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 3 index: 37. (index mismatch)
I20251026 09:10:43.264659 7189 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:43.266054 6604 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 3 index: 37. (index mismatch)
I20251026 09:10:43.266371 7208 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:43.295485 7222 raft_consensus.cc:1064] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: attempting to promote NON_VOTER 26c251f040b9472a92191f35891e8eed to VOTER
I20251026 09:10:43.295816 7222 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:43.296427 6938 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 LEARNER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch)
I20251026 09:10:43.296479 6533 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch)
I20251026 09:10:43.296559 6604 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch)
I20251026 09:10:43.296613 7189 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:43.296700 7222 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:43.296834 7222 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:43.297675 7189 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index 37 to 38, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:43.297793 6938 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index 37 to 38, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:43.297833 6605 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index 37 to 38, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:43.298913 6428 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index 37 to 38, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:43.298988 6533 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index 37 to 38, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:43.300511 6681 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:43.300899 6938 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20251026 09:10:43.301005 6533 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20251026 09:10:43.301103 7222 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:43.301220 7208 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:43.301848 7222 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index 38 to 39, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:43.301972 6938 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 38 to 39, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:43.302985 6533 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 38 to 39, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:43.303046 6416 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet ad759f01ba6c4154968854b4b9378651 with cas_config_opid_index 38: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:43.303121 6428 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 38 to 39, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New cstate: current_term: 1 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:43.304016 6596 tablet_service.cc:1552] Processing DeleteTablet for tablet ad759f01ba6c4154968854b4b9378651 with delete_type TABLET_DATA_TOMBSTONED (TS cfcab08da3db40e1974902b787ef6361 not found in new config with opid_index 39) from {username='slave'} at 127.0.0.1:47308
I20251026 09:10:43.304369 7228 tablet_replica.cc:333] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:10:43.304447 7228 raft_consensus.cc:2243] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:43.304550 7228 raft_consensus.cc:2272] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:43.304759 7228 ts_tablet_manager.cc:1916] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:43.306015 7228 ts_tablet_manager.cc:1929] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.38
I20251026 09:10:43.306077 7228 log.cc:1199] T ad759f01ba6c4154968854b4b9378651 P cfcab08da3db40e1974902b787ef6361: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/ad759f01ba6c4154968854b4b9378651
I20251026 09:10:43.306457 6417 catalog_manager.cc:4985] TS cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631): tablet ad759f01ba6c4154968854b4b9378651 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:43.433893 6684 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:43.434197 7161 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:43.435425 6605 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:43.435662 7161 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:43.662784 6605 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20251026 09:10:43.663103 7161 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251026 09:10:43.664290 6684 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 2 index: 36. (index mismatch)
I20251026 09:10:43.664630 7214 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251026 09:10:44.304921 6681 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:44.305559 6938 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
I20251026 09:10:44.305866 7236 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:44.305909 6533 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
I20251026 09:10:44.306125 7237 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:44.306886 7236 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.306998 6938 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.307687 6533 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.308876 6533 consensus_queue.cc:237] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:44.309022 6430 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 39 to 40, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New cstate: current_term: 1 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.309434 6605 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
W20251026 09:10:44.309518 6499 consensus_peers.cc:597] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c -> Peer 485b30c027fe4845b029de91594146fd (127.3.14.6:35539): Couldn't send request to peer 485b30c027fe4845b029de91594146fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac1d836cf54d42c39eb2d8279fd17f90. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:44.309518 6681 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
I20251026 09:10:44.309635 7214 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:44.309850 7214 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:44.310194 7234 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
W20251026 09:10:44.310497 6647 consensus_peers.cc:597] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad759f01ba6c4154968854b4b9378651. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:44.310487 6681 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:44.311206 6605 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:44.311779 6681 consensus_queue.cc:237] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 3.37, Last appended by leader: 36, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:44.312252 6533 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:44.312482 6605 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:44.312399 6427 catalog_manager.cc:5649] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index -1 to 38, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New cstate: current_term: 2 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:44.312419 7236 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
W20251026 09:10:44.312755 6650 consensus_peers.cc:597] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841): Couldn't send request to peer 26c251f040b9472a92191f35891e8eed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e002d8e9fb004210be27d87c5a23f9c3. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:44.312846 7237 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:44.313699 7189 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.38: config changed from index -1 to 38, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:44.313853 6605 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Committing config change with OpId 3.38: config changed from index -1 to 38, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:44.314775 6533 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Committing config change with OpId 3.38: config changed from index -1 to 38, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:44.315127 6429 catalog_manager.cc:5649] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index -1 to 38, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:44.318396 6605 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:44.319062 6681 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
I20251026 09:10:44.319221 6533 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
I20251026 09:10:44.319273 7255 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:44.319445 7254 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:44.320055 7255 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.320173 6681 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.321367 6429 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index -1 to 38, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.321794 6533 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
W20251026 09:10:44.322782 6572 consensus_peers.cc:597] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21bd52ca927842ccab72fc70494128d0. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:44.322817 6532 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 35, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:44.323369 6605 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:44.323580 7214 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:44.323633 6681 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:44.323860 7214 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:44.324309 7161 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Committing config change with OpId 2.37: config changed from index -1 to 37, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.324441 6605 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index -1 to 37, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.325568 6429 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index -1 to 37, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New cstate: current_term: 2 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:44.325703 6605 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 35, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:44.325789 6681 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index -1 to 37, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:44.327196 6532 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
W20251026 09:10:44.327293 6496 consensus_peers.cc:597] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bb5819b36e244d4b0ab2c0e110e51e9. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:44.327503 6681 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:44.327490 7255 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:44.327764 7261 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:44.328492 7261 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.37: config changed from index -1 to 37, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:44.328658 6681 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index -1 to 37, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:44.330371 6429 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index -1 to 37, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:44.330971 6533 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.37: config changed from index -1 to 37, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
W20251026 09:10:44.331579 6575 consensus_peers.cc:597] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 -> Peer 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841): Couldn't send request to peer 26c251f040b9472a92191f35891e8eed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff5ceda9af754029a9b5ac636a17ea70. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:44.699934 7265 ts_tablet_manager.cc:933] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: Initiating tablet copy from peer 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705)
I20251026 09:10:44.700183 7265 tablet_copy_client.cc:323] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.1:33705
I20251026 09:10:44.703409 6543 tablet_copy_service.cc:140] P 3ed603c643ac44ca9908b171687d7d5c: Received BeginTabletCopySession request for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 from peer 90365b2687854499bddaa63c08c0964f ({username='slave'} at 127.0.0.1:58756)
I20251026 09:10:44.703497 6543 tablet_copy_service.cc:161] P 3ed603c643ac44ca9908b171687d7d5c: Beginning new tablet copy session on tablet 4bb5819b36e244d4b0ab2c0e110e51e9 from peer 90365b2687854499bddaa63c08c0964f at {username='slave'} at 127.0.0.1:58756: session id = 90365b2687854499bddaa63c08c0964f-4bb5819b36e244d4b0ab2c0e110e51e9
I20251026 09:10:44.704108 6543 tablet_copy_source_session.cc:215] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:44.704511 7265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb5819b36e244d4b0ab2c0e110e51e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:44.706030 7265 tablet_copy_client.cc:806] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:44.706164 7265 tablet_copy_client.cc:670] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:44.706715 7266 tablet_copy_client.cc:1025] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:44.761240 7269 ts_tablet_manager.cc:933] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: Initiating tablet copy from peer 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705)
I20251026 09:10:44.761462 7269 tablet_copy_client.cc:323] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.1:33705
I20251026 09:10:44.764494 6543 tablet_copy_service.cc:140] P 3ed603c643ac44ca9908b171687d7d5c: Received BeginTabletCopySession request for tablet ac1d836cf54d42c39eb2d8279fd17f90 from peer 485b30c027fe4845b029de91594146fd ({username='slave'} at 127.0.0.1:58766)
I20251026 09:10:44.764617 6543 tablet_copy_service.cc:161] P 3ed603c643ac44ca9908b171687d7d5c: Beginning new tablet copy session on tablet ac1d836cf54d42c39eb2d8279fd17f90 from peer 485b30c027fe4845b029de91594146fd at {username='slave'} at 127.0.0.1:58766: session id = 485b30c027fe4845b029de91594146fd-ac1d836cf54d42c39eb2d8279fd17f90
I20251026 09:10:44.765200 6543 tablet_copy_source_session.cc:215] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:44.765553 7269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1d836cf54d42c39eb2d8279fd17f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:44.766788 7272 ts_tablet_manager.cc:933] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: Initiating tablet copy from peer cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:44.766916 7269 tablet_copy_client.cc:806] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:44.767069 7272 tablet_copy_client.cc:323] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42631
I20251026 09:10:44.767089 7269 tablet_copy_client.cc:670] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:44.767935 7270 tablet_copy_client.cc:1025] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:44.770267 6618 tablet_copy_service.cc:140] P cfcab08da3db40e1974902b787ef6361: Received BeginTabletCopySession request for tablet ff5ceda9af754029a9b5ac636a17ea70 from peer 26c251f040b9472a92191f35891e8eed ({username='slave'} at 127.0.0.1:44500)
I20251026 09:10:44.770345 6618 tablet_copy_service.cc:161] P cfcab08da3db40e1974902b787ef6361: Beginning new tablet copy session on tablet ff5ceda9af754029a9b5ac636a17ea70 from peer 26c251f040b9472a92191f35891e8eed at {username='slave'} at 127.0.0.1:44500: session id = 26c251f040b9472a92191f35891e8eed-ff5ceda9af754029a9b5ac636a17ea70
I20251026 09:10:44.770944 6618 tablet_copy_source_session.cc:215] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:44.771245 7272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5ceda9af754029a9b5ac636a17ea70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:44.772740 7272 tablet_copy_client.cc:806] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:44.772920 7272 tablet_copy_client.cc:670] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:44.773509 7273 tablet_copy_client.cc:1025] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:44.777283 7275 ts_tablet_manager.cc:933] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: Initiating tablet copy from peer cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:44.777524 7275 tablet_copy_client.cc:323] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42631
I20251026 09:10:44.781175 6618 tablet_copy_service.cc:140] P cfcab08da3db40e1974902b787ef6361: Received BeginTabletCopySession request for tablet 21bd52ca927842ccab72fc70494128d0 from peer 90365b2687854499bddaa63c08c0964f ({username='slave'} at 127.0.0.1:44510)
I20251026 09:10:44.781273 6618 tablet_copy_service.cc:161] P cfcab08da3db40e1974902b787ef6361: Beginning new tablet copy session on tablet 21bd52ca927842ccab72fc70494128d0 from peer 90365b2687854499bddaa63c08c0964f at {username='slave'} at 127.0.0.1:44510: session id = 90365b2687854499bddaa63c08c0964f-21bd52ca927842ccab72fc70494128d0
I20251026 09:10:44.782115 6618 tablet_copy_source_session.cc:215] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:44.782462 7275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bd52ca927842ccab72fc70494128d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:44.783823 7275 tablet_copy_client.cc:806] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:44.783964 7275 tablet_copy_client.cc:670] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:44.784499 7276 tablet_copy_client.cc:1025] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:44.833002 7277 ts_tablet_manager.cc:933] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: Initiating tablet copy from peer 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:44.833264 7277 tablet_copy_client.cc:323] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:39689
I20251026 09:10:44.836793 6694 tablet_copy_service.cc:140] P 6d20fbac1f1243a0b383c9383a1d0a70: Received BeginTabletCopySession request for tablet ad759f01ba6c4154968854b4b9378651 from peer 90365b2687854499bddaa63c08c0964f ({username='slave'} at 127.0.0.1:44960)
I20251026 09:10:44.836885 6694 tablet_copy_service.cc:161] P 6d20fbac1f1243a0b383c9383a1d0a70: Beginning new tablet copy session on tablet ad759f01ba6c4154968854b4b9378651 from peer 90365b2687854499bddaa63c08c0964f at {username='slave'} at 127.0.0.1:44960: session id = 90365b2687854499bddaa63c08c0964f-ad759f01ba6c4154968854b4b9378651
I20251026 09:10:44.837433 6694 tablet_copy_source_session.cc:215] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:44.837802 7277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad759f01ba6c4154968854b4b9378651. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:44.839697 7277 tablet_copy_client.cc:806] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:44.839840 7277 tablet_copy_client.cc:670] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:44.840528 7278 tablet_copy_client.cc:1025] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:44.849573 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:44.850065 6533 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "22a8d92b27b14f89a4728a8f50db932b"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:44.850160 6533 raft_consensus.cc:606] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:10:44.850493 6533 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:44.850584 6533 raft_consensus.cc:606] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Received request to transfer leadership to TS 6d20fbac1f1243a0b383c9383a1d0a70
I20251026 09:10:44.850881 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ad759f01ba6c4154968854b4b9378651"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "26c251f040b9472a92191f35891e8eed"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:44.850960 6684 raft_consensus.cc:606] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Received request to transfer leadership to TS 26c251f040b9472a92191f35891e8eed
I20251026 09:10:44.851248 6605 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:10:44.851318 6605 raft_consensus.cc:606] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:44.851514 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251026 09:10:44.851590 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:44.895800 7280 ts_tablet_manager.cc:933] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: Initiating tablet copy from peer 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:44.896047 7280 tablet_copy_client.cc:323] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:39689
I20251026 09:10:44.896306 6694 tablet_copy_service.cc:140] P 6d20fbac1f1243a0b383c9383a1d0a70: Received BeginTabletCopySession request for tablet e002d8e9fb004210be27d87c5a23f9c3 from peer 26c251f040b9472a92191f35891e8eed ({username='slave'} at 127.0.0.1:44952)
I20251026 09:10:44.896374 6694 tablet_copy_service.cc:161] P 6d20fbac1f1243a0b383c9383a1d0a70: Beginning new tablet copy session on tablet e002d8e9fb004210be27d87c5a23f9c3 from peer 26c251f040b9472a92191f35891e8eed at {username='slave'} at 127.0.0.1:44952: session id = 26c251f040b9472a92191f35891e8eed-e002d8e9fb004210be27d87c5a23f9c3
I20251026 09:10:44.896947 6694 tablet_copy_source_session.cc:215] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:44.897310 7280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e002d8e9fb004210be27d87c5a23f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:44.898372 7280 tablet_copy_client.cc:806] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:44.898486 7280 tablet_copy_client.cc:670] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:44.899092 7281 tablet_copy_client.cc:1025] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:44.950933 7261 raft_consensus.cc:993] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:44.951061 7261 raft_consensus.cc:1081] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:44.951241 6533 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44258
I20251026 09:10:44.951363 6533 raft_consensus.cc:493] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:44.951404 6533 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:44.952270 6533 raft_consensus.cc:515] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:44.952536 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:44.952584 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:44.952544 6533 leader_election.cc:290] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 3 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:44.952675 6681 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:44.952629 6605 raft_consensus.cc:3055] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:10:44.952729 6605 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:44.952786 6605 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:44.952899 6605 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:44.953402 6681 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 3.
I20251026 09:10:44.953505 6605 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 3.
I20251026 09:10:44.953614 6498 leader_election.cc:304] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:44.953724 7161 raft_consensus.cc:2804] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:10:44.953903 7161 raft_consensus.cc:697] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:44.953971 7161 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:44.954528 6427 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 2 to 3, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 3 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:45.023851 7283 raft_consensus.cc:993] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:45.023990 7283 raft_consensus.cc:1081] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:45.024166 6605 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "22a8d92b27b14f89a4728a8f50db932b"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47314
I20251026 09:10:45.024279 6605 raft_consensus.cc:493] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:45.024318 6605 raft_consensus.cc:3060] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:45.025256 6605 raft_consensus.cc:515] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:45.025441 6605 leader_election.cc:290] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:45.025599 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22a8d92b27b14f89a4728a8f50db932b" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:45.025573 6532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "22a8d92b27b14f89a4728a8f50db932b" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:45.025665 6681 raft_consensus.cc:3060] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:45.025672 6532 raft_consensus.cc:3055] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:45.025715 6532 raft_consensus.cc:740] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:45.025774 6532 consensus_queue.cc:260] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:45.025857 6532 raft_consensus.cc:3060] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:45.026602 6532 raft_consensus.cc:2468] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:45.026602 6681 raft_consensus.cc:2468] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:45.026832 6572 leader_election.cc:304] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:45.026964 7282 raft_consensus.cc:2804] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:45.027025 7282 raft_consensus.cc:697] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:45.027084 7282 consensus_queue.cc:237] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:45.027920 6427 catalog_manager.cc:5649] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 1 to 2, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:45.277447 7161 raft_consensus.cc:993] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:45.277549 7161 raft_consensus.cc:1081] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:45.277716 6681 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33632
I20251026 09:10:45.277791 6681 raft_consensus.cc:493] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:45.277818 6681 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:45.278642 6681 raft_consensus.cc:515] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:45.278826 6681 leader_election.cc:290] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:45.278959 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:45.279033 6604 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:45.279007 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bb5819b36e244d4b0ab2c0e110e51e9" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:45.279114 6533 raft_consensus.cc:3055] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:10:45.279141 6533 raft_consensus.cc:740] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:45.279196 6533 consensus_queue.cc:260] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:45.279292 6533 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:45.279742 6604 raft_consensus.cc:2468] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:45.279897 6533 raft_consensus.cc:2468] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:45.279901 6650 leader_election.cc:304] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:45.280072 7288 raft_consensus.cc:2804] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:10:45.280140 7288 raft_consensus.cc:697] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:45.280249 7288 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:45.280887 6427 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 2 to 3, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:45.328012 7237 raft_consensus.cc:993] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower 26c251f040b9472a92191f35891e8eed to start an election
I20251026 09:10:45.328152 7288 raft_consensus.cc:1081] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Signalling peer 26c251f040b9472a92191f35891e8eed to start an election
I20251026 09:10:45.328379 6938 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ad759f01ba6c4154968854b4b9378651"
dest_uuid: "26c251f040b9472a92191f35891e8eed"
from {username='slave'} at 127.0.0.1:36790
I20251026 09:10:45.328467 6938 raft_consensus.cc:493] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:45.328531 6938 raft_consensus.cc:3060] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:45.329386 6938 raft_consensus.cc:515] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:45.329586 6938 leader_election.cc:290] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [CANDIDATE]: Term 2 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:45.329741 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad759f01ba6c4154968854b4b9378651" candidate_uuid: "26c251f040b9472a92191f35891e8eed" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:45.329838 6681 raft_consensus.cc:3055] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:45.329871 6681 raft_consensus.cc:740] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:45.329921 6681 consensus_queue.cc:260] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:45.330017 6681 raft_consensus.cc:3060] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:45.330651 6681 raft_consensus.cc:2468] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c251f040b9472a92191f35891e8eed in term 2.
I20251026 09:10:45.330857 6903 leader_election.cc:304] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c251f040b9472a92191f35891e8eed, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:45.331044 7292 raft_consensus.cc:2804] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:45.331192 7292 raft_consensus.cc:697] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 LEADER]: Becoming Leader. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Running, Role: LEADER
I20251026 09:10:45.331444 7292 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:45.332072 6427 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed reported cstate change: term changed from 1 to 2, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to 26c251f040b9472a92191f35891e8eed (127.3.14.4). New cstate: current_term: 2 leader_uuid: "26c251f040b9472a92191f35891e8eed" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:45.333267 6532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad759f01ba6c4154968854b4b9378651" candidate_uuid: "26c251f040b9472a92191f35891e8eed" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:45.333350 6532 raft_consensus.cc:3060] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:45.334120 6532 raft_consensus.cc:2468] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c251f040b9472a92191f35891e8eed in term 2.
I20251026 09:10:45.411772 6684 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:45.412036 7297 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:45.413326 6533 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:45.413594 7297 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
W20251026 09:10:45.500533 6499 consensus_peers.cc:597] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c -> Peer 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841): Couldn't send request to peer 26c251f040b9472a92191f35891e8eed. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:45.533872 6684 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:45.534132 7161 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:45.535627 6604 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:45.535830 7283 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:45.686491 6605 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:45.686740 7298 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:45.687798 6533 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:45.688021 7237 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
W20251026 09:10:45.688176 6647 consensus_peers.cc:597] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:45.736438 6533 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch)
I20251026 09:10:45.736691 7292 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:45.738565 6681 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch)
I20251026 09:10:45.738807 7311 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20251026 09:10:45.741917 6901 consensus_peers.cc:597] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:46.706995 7265 tablet_copy_client.cc:538] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:46.707923 7265 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: Bootstrap starting.
I20251026 09:10:46.712635 7265 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=37 overwritten=0 applied=37 ignored=0} inserts{seen=140 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:46.712875 7265 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: Bootstrap complete.
I20251026 09:10:46.713008 7265 ts_tablet_manager.cc:1403] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 09:10:46.713196 7265 raft_consensus.cc:359] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:46.713292 7265 raft_consensus.cc:740] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 90365b2687854499bddaa63c08c0964f, State: Initialized, Role: LEARNER
I20251026 09:10:46.713332 7265 consensus_queue.cc:260] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:46.713515 7265 ts_tablet_manager.cc:1434] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20251026 09:10:46.713752 6543 tablet_copy_service.cc:342] P 3ed603c643ac44ca9908b171687d7d5c: Request end of tablet copy session 90365b2687854499bddaa63c08c0964f-4bb5819b36e244d4b0ab2c0e110e51e9 received from {username='slave'} at 127.0.0.1:58756
I20251026 09:10:46.713827 6543 tablet_copy_service.cc:434] P 3ed603c643ac44ca9908b171687d7d5c: ending tablet copy session 90365b2687854499bddaa63c08c0964f-4bb5819b36e244d4b0ab2c0e110e51e9 on tablet 4bb5819b36e244d4b0ab2c0e110e51e9 with peer 90365b2687854499bddaa63c08c0964f
I20251026 09:10:46.768172 7269 tablet_copy_client.cc:538] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:46.769122 7269 tablet_bootstrap.cc:492] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: Bootstrap starting.
I20251026 09:10:46.773743 7272 tablet_copy_client.cc:538] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:46.774557 7269 tablet_bootstrap.cc:492] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=142 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:46.774791 7269 tablet_bootstrap.cc:492] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: Bootstrap complete.
I20251026 09:10:46.774807 7272 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: Bootstrap starting.
I20251026 09:10:46.774973 7269 ts_tablet_manager.cc:1403] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20251026 09:10:46.775156 7269 raft_consensus.cc:359] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:46.775244 7269 raft_consensus.cc:740] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 485b30c027fe4845b029de91594146fd, State: Initialized, Role: LEARNER
I20251026 09:10:46.775279 7269 consensus_queue.cc:260] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 2.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:46.775470 7269 ts_tablet_manager.cc:1434] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:46.776635 6543 tablet_copy_service.cc:342] P 3ed603c643ac44ca9908b171687d7d5c: Request end of tablet copy session 485b30c027fe4845b029de91594146fd-ac1d836cf54d42c39eb2d8279fd17f90 received from {username='slave'} at 127.0.0.1:58766
I20251026 09:10:46.776677 6543 tablet_copy_service.cc:434] P 3ed603c643ac44ca9908b171687d7d5c: ending tablet copy session 485b30c027fe4845b029de91594146fd-ac1d836cf54d42c39eb2d8279fd17f90 on tablet ac1d836cf54d42c39eb2d8279fd17f90 with peer 485b30c027fe4845b029de91594146fd
I20251026 09:10:46.779711 7272 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: Bootstrap replayed 1/1 log segments. Stats: ops{read=37 overwritten=0 applied=37 ignored=0} inserts{seen=142 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:46.780082 7272 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: Bootstrap complete.
I20251026 09:10:46.780254 7272 ts_tablet_manager.cc:1403] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.001s
I20251026 09:10:46.780454 7272 raft_consensus.cc:359] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:46.780545 7272 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Initialized, Role: LEARNER
I20251026 09:10:46.780581 7272 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:46.780683 7272 ts_tablet_manager.cc:1434] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:46.780880 6618 tablet_copy_service.cc:342] P cfcab08da3db40e1974902b787ef6361: Request end of tablet copy session 26c251f040b9472a92191f35891e8eed-ff5ceda9af754029a9b5ac636a17ea70 received from {username='slave'} at 127.0.0.1:44500
I20251026 09:10:46.780946 6618 tablet_copy_service.cc:434] P cfcab08da3db40e1974902b787ef6361: ending tablet copy session 26c251f040b9472a92191f35891e8eed-ff5ceda9af754029a9b5ac636a17ea70 on tablet ff5ceda9af754029a9b5ac636a17ea70 with peer 26c251f040b9472a92191f35891e8eed
I20251026 09:10:46.784727 7275 tablet_copy_client.cc:538] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:46.785710 7275 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: Bootstrap starting.
I20251026 09:10:46.790467 7275 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=143 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:46.790752 7275 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: Bootstrap complete.
I20251026 09:10:46.790899 7275 ts_tablet_manager.cc:1403] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251026 09:10:46.791087 7275 raft_consensus.cc:359] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:46.791184 7275 raft_consensus.cc:740] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 90365b2687854499bddaa63c08c0964f, State: Initialized, Role: LEARNER
I20251026 09:10:46.791225 7275 consensus_queue.cc:260] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 2.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:46.791396 7275 ts_tablet_manager.cc:1434] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: Time spent starting tablet: real 0.000s user 0.004s sys 0.000s
I20251026 09:10:46.791597 6618 tablet_copy_service.cc:342] P cfcab08da3db40e1974902b787ef6361: Request end of tablet copy session 90365b2687854499bddaa63c08c0964f-21bd52ca927842ccab72fc70494128d0 received from {username='slave'} at 127.0.0.1:44510
I20251026 09:10:46.791671 6618 tablet_copy_service.cc:434] P cfcab08da3db40e1974902b787ef6361: ending tablet copy session 90365b2687854499bddaa63c08c0964f-21bd52ca927842ccab72fc70494128d0 on tablet 21bd52ca927842ccab72fc70494128d0 with peer 90365b2687854499bddaa63c08c0964f
I20251026 09:10:46.840746 7277 tablet_copy_client.cc:538] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:46.840914 7014 raft_consensus.cc:1217] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.37->[2.38-2.38] Dedup: 2.38->[]
I20251026 09:10:46.841897 7277 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: Bootstrap starting.
I20251026 09:10:46.847688 7277 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=148 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:46.847973 7277 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: Bootstrap complete.
I20251026 09:10:46.848176 7277 ts_tablet_manager.cc:1403] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20251026 09:10:46.848380 7277 raft_consensus.cc:359] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:46.848459 7277 raft_consensus.cc:740] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90365b2687854499bddaa63c08c0964f, State: Initialized, Role: LEARNER
I20251026 09:10:46.848505 7277 consensus_queue.cc:260] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:46.848652 7277 ts_tablet_manager.cc:1434] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:10:46.848863 6694 tablet_copy_service.cc:342] P 6d20fbac1f1243a0b383c9383a1d0a70: Request end of tablet copy session 90365b2687854499bddaa63c08c0964f-ad759f01ba6c4154968854b4b9378651 received from {username='slave'} at 127.0.0.1:44960
I20251026 09:10:46.848917 6694 tablet_copy_service.cc:434] P 6d20fbac1f1243a0b383c9383a1d0a70: ending tablet copy session 90365b2687854499bddaa63c08c0964f-ad759f01ba6c4154968854b4b9378651 on tablet ad759f01ba6c4154968854b4b9378651 with peer 90365b2687854499bddaa63c08c0964f
I20251026 09:10:46.851733 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:46.852196 6604 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:10:46.852283 6604 raft_consensus.cc:606] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Received request to transfer leadership to TS 6d20fbac1f1243a0b383c9383a1d0a70
I20251026 09:10:46.852735 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:46.852818 6684 raft_consensus.cc:606] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:46.853245 6532 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:46.853322 6532 raft_consensus.cc:606] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:10:46.853498 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20251026 09:10:46.853577 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:46.859018 7261 raft_consensus.cc:993] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:46.859130 7261 raft_consensus.cc:1081] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:46.859301 6684 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33638
I20251026 09:10:46.859426 6684 raft_consensus.cc:493] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:46.859465 6684 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:46.860241 6684 raft_consensus.cc:515] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:46.860380 6684 leader_election.cc:290] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:46.860499 6532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:46.860587 6532 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:46.860548 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:46.860647 6604 raft_consensus.cc:3055] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:10:46.860682 6604 raft_consensus.cc:740] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:46.860751 6604 consensus_queue.cc:260] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 2.38, Last appended by leader: 38, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:46.860872 6604 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:46.861290 6532 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:46.861495 6647 leader_election.cc:304] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:46.861512 6604 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:46.861603 7288 raft_consensus.cc:2804] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:10:46.861723 7288 raft_consensus.cc:697] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:46.861791 7288 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 2.38, Last appended by leader: 38, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:46.862298 6429 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 2 to 3, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:46.865182 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33620
I20251026 09:10:46.865274 6684 raft_consensus.cc:606] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Received request to transfer leadership
I20251026 09:10:46.867723 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33620
I20251026 09:10:46.867947 6684 raft_consensus.cc:606] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Received request to transfer leadership
W20251026 09:10:46.868173 6416 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 21bd52ca927842ccab72fc70494128d0 already in progress [suppressed 1 similar messages]
I20251026 09:10:46.868569 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33620
I20251026 09:10:46.868641 6684 raft_consensus.cc:606] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Received request to transfer leadership
W20251026 09:10:46.868918 6478 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 21bd52ca927842ccab72fc70494128d0 already in progress
W20251026 09:10:46.868980 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 21bd52ca927842ccab72fc70494128d0 already in progress
I20251026 09:10:46.871747 7014 raft_consensus.cc:3060] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Advancing to term 3
I20251026 09:10:46.899399 7280 tablet_copy_client.cc:538] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:46.900280 7280 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: Bootstrap starting.
I20251026 09:10:46.904542 7280 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=148 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:46.904835 7280 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: Bootstrap complete.
I20251026 09:10:46.904976 7280 ts_tablet_manager.cc:1403] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 09:10:46.905330 7280 raft_consensus.cc:359] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:46.905439 7280 raft_consensus.cc:740] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Initialized, Role: LEARNER
I20251026 09:10:46.905475 7280 consensus_queue.cc:260] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:46.905625 7280 ts_tablet_manager.cc:1434] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 09:10:46.905807 6694 tablet_copy_service.cc:342] P 6d20fbac1f1243a0b383c9383a1d0a70: Request end of tablet copy session 26c251f040b9472a92191f35891e8eed-e002d8e9fb004210be27d87c5a23f9c3 received from {username='slave'} at 127.0.0.1:44952
I20251026 09:10:46.905860 6694 tablet_copy_service.cc:434] P 6d20fbac1f1243a0b383c9383a1d0a70: ending tablet copy session 26c251f040b9472a92191f35891e8eed-e002d8e9fb004210be27d87c5a23f9c3 on tablet e002d8e9fb004210be27d87c5a23f9c3 with peer 26c251f040b9472a92191f35891e8eed
I20251026 09:10:47.028210 7283 raft_consensus.cc:993] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:47.028321 7283 raft_consensus.cc:1081] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:47.028537 6604 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47314
I20251026 09:10:47.028656 6604 raft_consensus.cc:493] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:47.028707 6604 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:47.029582 6604 raft_consensus.cc:515] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:47.029716 6604 leader_election.cc:290] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:47.029951 6532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:47.029968 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:47.030079 6532 raft_consensus.cc:3055] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:47.030103 6681 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:47.030153 6532 raft_consensus.cc:740] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:47.030277 6532 consensus_queue.cc:260] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:47.030367 6532 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:47.031014 6681 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:47.031164 6574 leader_election.cc:304] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:47.031256 7261 raft_consensus.cc:2804] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:47.031267 6532 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 2.
I20251026 09:10:47.031378 7261 raft_consensus.cc:697] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:47.031658 7261 consensus_queue.cc:237] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:47.032256 6429 catalog_manager.cc:5649] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 1 to 2, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:47.050230 6938 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 2 LEARNER]: Advancing to term 3
I20251026 09:10:47.065299 7288 raft_consensus.cc:993] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:47.065452 7322 raft_consensus.cc:1081] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:47.065712 6532 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44252
I20251026 09:10:47.065804 6532 raft_consensus.cc:493] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:47.065836 6532 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:47.066766 6532 raft_consensus.cc:515] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:47.066924 6532 leader_election.cc:290] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 4 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:47.067036 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:47.067116 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:47.067121 6604 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:47.067253 6681 raft_consensus.cc:3055] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 09:10:47.067281 6681 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:47.067384 6681 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:47.067478 6681 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:47.068159 6604 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 4.
I20251026 09:10:47.068197 6681 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 4.
I20251026 09:10:47.068394 6499 leader_election.cc:304] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:47.068509 7283 raft_consensus.cc:2804] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Leader election won for term 4
I20251026 09:10:47.068579 7283 raft_consensus.cc:697] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:47.068634 7283 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:47.069324 6429 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 3 to 4, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 4 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:47.122182 7014 raft_consensus.cc:3060] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 1 LEARNER]: Advancing to term 2
I20251026 09:10:47.135320 6938 raft_consensus.cc:1217] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 LEARNER]: Deduplicated request from leader. Original: 3.37->[3.38-3.38] Dedup: 3.38->[]
I20251026 09:10:47.144510 7089 raft_consensus.cc:1217] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.37->[2.38-2.38] Dedup: 2.38->[]
I20251026 09:10:47.238731 6604 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.238988 7322 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.239923 7288 raft_consensus.cc:993] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:47.240005 7288 raft_consensus.cc:1081] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:47.240227 6605 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47312
I20251026 09:10:47.240314 6605 raft_consensus.cc:493] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:47.240347 6605 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:47.240586 7014 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Advancing to term 3
I20251026 09:10:47.240974 6532 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.241371 7014 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 3 LEARNER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.241223 7341 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.241675 7288 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.241740 6605 raft_consensus.cc:515] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:47.241876 6605 leader_election.cc:290] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:47.242152 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:47.242223 6684 raft_consensus.cc:3055] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 09:10:47.242246 6684 raft_consensus.cc:740] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:47.242296 6684 consensus_queue.cc:260] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 3.39, Last appended by leader: 39, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:47.242451 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:47.242730 6684 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:47.243470 6684 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 4.
I20251026 09:10:47.243662 6574 leader_election.cc:304] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters: 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:47.243824 7261 raft_consensus.cc:2804] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 09:10:47.243913 7261 raft_consensus.cc:697] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:47.243992 7261 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 3.39, Last appended by leader: 39, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:47.244366 6427 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 3 to 4, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:47.274330 7322 raft_consensus.cc:1064] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: attempting to promote NON_VOTER 90365b2687854499bddaa63c08c0964f to VOTER
I20251026 09:10:47.274616 7322 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 37, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:47.275368 6533 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.275386 7014 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 LEARNER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.275424 6605 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.275712 7347 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.275954 7288 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.276054 7341 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.276896 7322 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.39: config changed from index 37 to 39, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.277063 6605 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 37 to 39, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.277668 6533 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 37 to 39, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.278239 6429 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 37 to 39, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:47.279203 7014 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 37 to 39, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.280392 6681 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 3.39, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:47.280824 7014 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251026 09:10:47.280946 6533 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251026 09:10:47.280997 7288 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.281152 7288 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.281579 7347 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.281731 7014 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.282613 6416 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:47.282807 6428 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f reported cstate change: config changed from index 39 to 40, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.283124 6532 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.284253 6596 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 with delete_type TABLET_DATA_TOMBSTONED (TS cfcab08da3db40e1974902b787ef6361 not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:47308
I20251026 09:10:47.284507 7350 tablet_replica.cc:333] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:10:47.284591 7350 raft_consensus.cc:2243] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:47.284693 7350 raft_consensus.cc:2272] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:47.284962 7350 ts_tablet_manager.cc:1916] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:47.286455 7350 ts_tablet_manager.cc:1929] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.39
I20251026 09:10:47.286523 7350 log.cc:1199] T 4bb5819b36e244d4b0ab2c0e110e51e9 P cfcab08da3db40e1974902b787ef6361: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/4bb5819b36e244d4b0ab2c0e110e51e9
I20251026 09:10:47.286906 6417 catalog_manager.cc:4985] TS cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631): tablet 4bb5819b36e244d4b0ab2c0e110e51e9 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:47.427599 6533 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:47.427917 7261 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:47.429529 6684 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:47.429899 7261 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:47.533762 7353 raft_consensus.cc:1064] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: attempting to promote NON_VOTER 26c251f040b9472a92191f35891e8eed to VOTER
I20251026 09:10:47.534065 7353 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 37, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:47.534618 6938 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 3 LEARNER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.534652 6605 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.534811 6684 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.534863 7283 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.535000 7332 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.535187 7283 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.536034 7353 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 LEADER]: Committing config change with OpId 3.39: config changed from index 37 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.536202 6605 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 37 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.536247 6938 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 37 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.537577 7347 raft_consensus.cc:1064] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: attempting to promote NON_VOTER 26c251f040b9472a92191f35891e8eed to VOTER
I20251026 09:10:47.537642 6684 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 37 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.537843 7347 consensus_queue.cc:237] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 36, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:47.538035 6429 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 37 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:47.538664 6532 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.538791 6938 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 LEARNER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.538841 6605 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 3 index: 39. (index mismatch)
I20251026 09:10:47.539057 7341 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.539156 7288 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.539291 7288 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.539850 6533 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 3.39, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:47.540272 6605 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251026 09:10:47.540249 6937 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251026 09:10:47.540549 7355 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.540568 7322 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.39: config changed from index 38 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.540638 7283 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.541002 6938 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 38 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.541296 7283 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 LEADER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.542219 6414 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet ff5ceda9af754029a9b5ac636a17ea70 with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:47.542255 6428 catalog_manager.cc:5649] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed reported cstate change: config changed from index 38 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.542804 6604 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.542819 6429 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New cstate: current_term: 3 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:47.543126 7355 raft_consensus.cc:1064] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c: attempting to promote NON_VOTER 485b30c027fe4845b029de91594146fd to VOTER
I20251026 09:10:47.543116 6937 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.543301 7355 consensus_queue.cc:237] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 2.38, Last appended by leader: 36, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:47.544032 7089 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 2 LEARNER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20251026 09:10:47.544149 6604 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20251026 09:10:47.544258 6684 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20251026 09:10:47.544332 6681 consensus_queue.cc:237] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 3.39, Last appended by leader: 36, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:47.544255 6665 tablet_service.cc:1552] Processing DeleteTablet for tablet ff5ceda9af754029a9b5ac636a17ea70 with delete_type TABLET_DATA_TOMBSTONED (TS 6d20fbac1f1243a0b383c9383a1d0a70 not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:33620
I20251026 09:10:47.544505 6605 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Committing config change with OpId 3.39: config changed from index 38 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.544710 7364 tablet_replica.cc:333] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:10:47.544904 7364 raft_consensus.cc:2243] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:47.544979 6937 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251026 09:10:47.545002 7364 raft_consensus.cc:2272] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:47.545243 7364 ts_tablet_manager.cc:1916] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:47.545233 7288 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.544364 7353 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.001s
I20251026 09:10:47.545596 7355 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.545744 7332 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.545957 6604 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20251026 09:10:47.546743 7355 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:47.546955 6605 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:47.547230 7364 ts_tablet_manager.cc:1929] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.39
I20251026 09:10:47.547271 7364 log.cc:1199] T ff5ceda9af754029a9b5ac636a17ea70 P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/ff5ceda9af754029a9b5ac636a17ea70
I20251026 09:10:47.547627 6416 catalog_manager.cc:4985] TS 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689): tablet ff5ceda9af754029a9b5ac636a17ea70 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:47.548028 6430 catalog_manager.cc:5649] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:47.548024 7089 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:47.548049 6681 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:47.548225 7288 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.548861 7288 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.549777 6416 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet e002d8e9fb004210be27d87c5a23f9c3 with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:47.549986 6533 consensus_queue.cc:237] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:47.550364 7089 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20251026 09:10:47.550376 6605 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20251026 09:10:47.550470 6937 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.551187 7353 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.551280 7332 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.551381 6428 catalog_manager.cc:5649] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed reported cstate change: config changed from index 39 to 40, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.551759 6604 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Committing config change with OpId 3.40: config changed from index 39 to 40, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:47.551955 7353 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:47.552112 6518 tablet_service.cc:1552] Processing DeleteTablet for tablet e002d8e9fb004210be27d87c5a23f9c3 with delete_type TABLET_DATA_TOMBSTONED (TS 3ed603c643ac44ca9908b171687d7d5c not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:47.552348 7368 tablet_replica.cc:333] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:10:47.552403 7368 raft_consensus.cc:2243] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:47.552438 7368 pending_rounds.cc:70] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: Trying to abort 1 pending ops.
I20251026 09:10:47.552464 7368 pending_rounds.cc:77] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: Aborting op as it isn't in flight: id { term: 3 index: 39 } timestamp: 7214980495514361856 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "e002d8e9fb004210be27d87c5a23f9c3" old_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } } new_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } } }
I20251026 09:10:47.552546 7368 raft_consensus.cc:2917] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Aborting config change with OpId 3.39: Aborted: Op aborted
I20251026 09:10:47.552565 7368 raft_consensus.cc:2889] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20251026 09:10:47.552585 7368 raft_consensus.cc:2272] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:47.552805 6414 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet ac1d836cf54d42c39eb2d8279fd17f90 with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:47.552989 6604 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:47.553045 6428 catalog_manager.cc:5649] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New cstate: current_term: 2 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:47.553802 6665 tablet_service.cc:1552] Processing DeleteTablet for tablet ac1d836cf54d42c39eb2d8279fd17f90 with delete_type TABLET_DATA_TOMBSTONED (TS 6d20fbac1f1243a0b383c9383a1d0a70 not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:33620
I20251026 09:10:47.553887 7364 tablet_replica.cc:333] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:10:47.554168 7364 raft_consensus.cc:2243] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:47.554360 7364 raft_consensus.cc:2272] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:47.554605 7368 ts_tablet_manager.cc:1916] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:47.554806 7089 raft_consensus.cc:2955] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:47.556159 7364 ts_tablet_manager.cc:1916] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:47.556305 7368 ts_tablet_manager.cc:1929] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.39
I20251026 09:10:47.556362 7368 log.cc:1199] T e002d8e9fb004210be27d87c5a23f9c3 P 3ed603c643ac44ca9908b171687d7d5c: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/e002d8e9fb004210be27d87c5a23f9c3
I20251026 09:10:47.556756 6414 catalog_manager.cc:4985] TS 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705): tablet e002d8e9fb004210be27d87c5a23f9c3 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:47.557646 7364 ts_tablet_manager.cc:1929] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.39
I20251026 09:10:47.557708 7364 log.cc:1199] T ac1d836cf54d42c39eb2d8279fd17f90 P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/ac1d836cf54d42c39eb2d8279fd17f90
I20251026 09:10:47.558085 6416 catalog_manager.cc:4985] TS 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689): tablet ac1d836cf54d42c39eb2d8279fd17f90 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:47.596421 7369 raft_consensus.cc:1064] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: attempting to promote NON_VOTER 90365b2687854499bddaa63c08c0964f to VOTER
I20251026 09:10:47.596697 7369 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 2.41, Last appended by leader: 40, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:47.597231 6681 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 2 index: 42. (index mismatch)
I20251026 09:10:47.597250 7014 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 2 index: 42. (index mismatch)
I20251026 09:10:47.597407 6533 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 2 index: 42. (index mismatch)
I20251026 09:10:47.597487 7293 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:47.597719 7371 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:47.597863 7369 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:47.598798 7369 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 LEADER]: Committing config change with OpId 2.42: config changed from index 40 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.598893 6681 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.42: config changed from index 40 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.599908 7014 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Committing config change with OpId 2.42: config changed from index 40 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.599943 6428 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 40 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "26c251f040b9472a92191f35891e8eed" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.600121 6533 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.42: config changed from index 40 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.601157 6937 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 2.42, Last appended by leader: 40, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:47.601717 6681 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 42. Preceding OpId from leader: term: 2 index: 43. (index mismatch)
I20251026 09:10:47.601761 7014 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 42. Preceding OpId from leader: term: 2 index: 43. (index mismatch)
I20251026 09:10:47.601928 7371 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:47.602011 7293 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:47.602540 7369 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 LEADER]: Committing config change with OpId 2.43: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.602674 6681 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.43: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.602782 7014 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Committing config change with OpId 2.43: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.603700 6477 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet ad759f01ba6c4154968854b4b9378651 with cas_config_opid_index 42: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:47.603811 6430 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f reported cstate change: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New cstate: current_term: 2 leader_uuid: "26c251f040b9472a92191f35891e8eed" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.604596 6518 tablet_service.cc:1552] Processing DeleteTablet for tablet ad759f01ba6c4154968854b4b9378651 with delete_type TABLET_DATA_TOMBSTONED (TS 3ed603c643ac44ca9908b171687d7d5c not found in new config with opid_index 43) from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:47.604723 7368 tablet_replica.cc:333] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:10:47.604818 7368 raft_consensus.cc:2243] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:47.605013 7368 raft_consensus.cc:2272] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:47.605300 7368 ts_tablet_manager.cc:1916] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:47.606663 7368 ts_tablet_manager.cc:1929] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.42
I20251026 09:10:47.606721 7368 log.cc:1199] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/ad759f01ba6c4154968854b4b9378651
I20251026 09:10:47.607061 6414 catalog_manager.cc:4985] TS 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705): tablet ad759f01ba6c4154968854b4b9378651 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:47.639762 6681 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 4 index: 39. (index mismatch)
I20251026 09:10:47.640079 7355 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.641238 6604 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 4 index: 39. (index mismatch)
I20251026 09:10:47.641659 7332 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:47.650923 6681 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 4 index: 40. (index mismatch)
I20251026 09:10:47.651221 7261 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.652351 6533 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:47.652356 7014 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 3 LEARNER]: Advancing to term 4
I20251026 09:10:47.653313 6533 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 4 index: 40. (index mismatch)
I20251026 09:10:47.653314 7014 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 4 LEARNER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 4 index: 40. (index mismatch)
I20251026 09:10:47.653559 7261 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.653661 7376 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:47.654350 7261 raft_consensus.cc:1064] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: attempting to promote NON_VOTER 90365b2687854499bddaa63c08c0964f to VOTER
I20251026 09:10:47.654531 7261 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 4.40, Last appended by leader: 39, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:47.654963 6681 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 40. Preceding OpId from leader: term: 4 index: 41. (index mismatch)
I20251026 09:10:47.655041 6533 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 40. Preceding OpId from leader: term: 4 index: 41. (index mismatch)
I20251026 09:10:47.655066 7014 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 4 LEARNER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 40. Preceding OpId from leader: term: 4 index: 41. (index mismatch)
I20251026 09:10:47.655246 7377 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:47.655349 7376 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:47.655485 7261 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:47.655978 7377 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Committing config change with OpId 4.41: config changed from index 38 to 41, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.656129 6533 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Committing config change with OpId 4.41: config changed from index 38 to 41, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.656217 6681 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Committing config change with OpId 4.41: config changed from index 38 to 41, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.657306 6430 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 38 to 41, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.657634 7014 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 4 FOLLOWER]: Committing config change with OpId 4.41: config changed from index 38 to 41, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.658716 6604 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 4.41, Last appended by leader: 39, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:47.659176 6533 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20251026 09:10:47.659224 7014 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20251026 09:10:47.659389 7377 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:47.659461 7261 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:47.659991 7377 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Committing config change with OpId 4.42: config changed from index 41 to 42, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.660105 7014 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 4 FOLLOWER]: Committing config change with OpId 4.42: config changed from index 41 to 42, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.660153 6533 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Committing config change with OpId 4.42: config changed from index 41 to 42, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.661199 6428 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f reported cstate change: config changed from index 41 to 42, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:47.661903 6417 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 21bd52ca927842ccab72fc70494128d0 with cas_config_opid_index 41: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:47.662245 6665 tablet_service.cc:1552] Processing DeleteTablet for tablet 21bd52ca927842ccab72fc70494128d0 with delete_type TABLET_DATA_TOMBSTONED (TS 6d20fbac1f1243a0b383c9383a1d0a70 not found in new config with opid_index 42) from {username='slave'} at 127.0.0.1:33620
I20251026 09:10:47.662353 7364 tablet_replica.cc:333] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:10:47.662431 7364 raft_consensus.cc:2243] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:47.662521 7364 raft_consensus.cc:2272] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:47.662755 7364 ts_tablet_manager.cc:1916] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:47.664250 7364 ts_tablet_manager.cc:1929] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.41
I20251026 09:10:47.664301 7364 log.cc:1199] T 21bd52ca927842ccab72fc70494128d0 P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/21bd52ca927842ccab72fc70494128d0
I20251026 09:10:47.664662 6416 catalog_manager.cc:4985] TS 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689): tablet 21bd52ca927842ccab72fc70494128d0 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:48.604957 6681 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:48.605648 7014 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
W20251026 09:10:48.605674 6650 consensus_peers.cc:597] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 485b30c027fe4845b029de91594146fd (127.3.14.6:35539): Couldn't send request to peer 485b30c027fe4845b029de91594146fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bb5819b36e244d4b0ab2c0e110e51e9. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:48.605722 6533 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251026 09:10:48.605866 7381 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:48.606025 7382 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:48.606848 7322 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:48.607055 7014 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:48.607395 6533 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:48.608614 6604 consensus_queue.cc:237] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:48.608630 6429 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f reported cstate change: config changed from index 40 to 41, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:48.609304 6681 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
W20251026 09:10:48.609365 6575 consensus_peers.cc:597] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 -> Peer 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841): Couldn't send request to peer 26c251f040b9472a92191f35891e8eed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22a8d92b27b14f89a4728a8f50db932b. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:48.609438 6533 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
I20251026 09:10:48.609606 7377 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:48.609704 7388 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:48.610622 7388 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:48.610738 6681 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:48.611604 6533 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:48.612109 6681 consensus_queue.cc:237] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 36, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:48.612123 6430 catalog_manager.cc:5649] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index -1 to 38, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:48.612641 6938 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
W20251026 09:10:48.612805 6647 consensus_peers.cc:597] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e002d8e9fb004210be27d87c5a23f9c3. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:48.612838 7381 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:48.613093 6604 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251026 09:10:48.613304 7381 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:48.613722 7381 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:48.613871 6938 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:48.614926 6428 catalog_manager.cc:5649] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed reported cstate change: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:48.615093 6605 consensus_queue.cc:237] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:48.615626 6681 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
I20251026 09:10:48.615715 6533 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 2 index: 38. (index mismatch)
I20251026 09:10:48.615816 7388 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
W20251026 09:10:48.616113 6575 consensus_peers.cc:597] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 -> Peer 485b30c027fe4845b029de91594146fd (127.3.14.6:35539): Couldn't send request to peer 485b30c027fe4845b029de91594146fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9723f81063e4f23bb2300f2c7bd50eb. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:48.616226 7388 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:48.616430 6605 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:48.617218 7388 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:48.617391 6681 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:48.618363 6429 catalog_manager.cc:5649] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index -1 to 38, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:48.618891 6533 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.39, Last appended by leader: 38, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:48.618920 6532 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.38: config changed from index -1 to 38, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:48.619349 6681 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 4 index: 40. (index mismatch)
I20251026 09:10:48.619459 6605 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 4 index: 40. (index mismatch)
W20251026 09:10:48.619496 6499 consensus_peers.cc:597] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c -> Peer 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841): Couldn't send request to peer 26c251f040b9472a92191f35891e8eed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6459af4a4180422a8eccf12fc862499e. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:48.619549 7379 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:48.619642 7332 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:48.620425 7332 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Committing config change with OpId 4.40: config changed from index -1 to 40, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:48.620515 6605 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Committing config change with OpId 4.40: config changed from index -1 to 40, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:48.620561 6681 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Committing config change with OpId 4.40: config changed from index -1 to 40, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:48.621776 6681 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:48.621819 6429 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index -1 to 40, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New cstate: current_term: 4 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:48.622284 7089 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251026 09:10:48.622285 6532 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251026 09:10:48.622480 7382 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20251026 09:10:48.622565 6647 consensus_peers.cc:597] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75fb5eb7df3b4c87bfdaa0b3f0efe601. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:48.622565 7322 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:48.623402 7381 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:48.623539 7089 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:48.624538 6532 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:48.624672 6430 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:48.853730 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:48.854246 6604 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "90365b2687854499bddaa63c08c0964f"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:10:48.854367 6604 raft_consensus.cc:606] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Received request to transfer leadership to TS 90365b2687854499bddaa63c08c0964f
I20251026 09:10:48.854801 6532 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:48.854880 6532 raft_consensus.cc:606] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Received request to transfer leadership to TS 6d20fbac1f1243a0b383c9383a1d0a70
I20251026 09:10:48.855178 6681 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:48.855247 6681 raft_consensus.cc:606] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:48.855615 6532 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "485b30c027fe4845b029de91594146fd"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:48.855685 6532 raft_consensus.cc:606] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Received request to transfer leadership to TS 485b30c027fe4845b029de91594146fd
I20251026 09:10:48.855995 6604 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:10:48.856071 6604 raft_consensus.cc:606] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:48.856248 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251026 09:10:48.856323 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:48.994444 7405 ts_tablet_manager.cc:933] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: Initiating tablet copy from peer 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:48.994648 7405 tablet_copy_client.cc:323] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:39689
I20251026 09:10:48.994880 6694 tablet_copy_service.cc:140] P 6d20fbac1f1243a0b383c9383a1d0a70: Received BeginTabletCopySession request for tablet e002d8e9fb004210be27d87c5a23f9c3 from peer 90365b2687854499bddaa63c08c0964f ({username='slave'} at 127.0.0.1:44960)
I20251026 09:10:48.994930 6694 tablet_copy_service.cc:161] P 6d20fbac1f1243a0b383c9383a1d0a70: Beginning new tablet copy session on tablet e002d8e9fb004210be27d87c5a23f9c3 from peer 90365b2687854499bddaa63c08c0964f at {username='slave'} at 127.0.0.1:44960: session id = 90365b2687854499bddaa63c08c0964f-e002d8e9fb004210be27d87c5a23f9c3
I20251026 09:10:48.995553 6694 tablet_copy_source_session.cc:215] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:48.995965 7405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e002d8e9fb004210be27d87c5a23f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:48.997488 7405 tablet_copy_client.cc:806] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:48.997705 7405 tablet_copy_client.cc:670] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:48.998270 7406 tablet_copy_client.cc:1025] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:49.000489 7407 ts_tablet_manager.cc:933] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: Initiating tablet copy from peer 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705)
I20251026 09:10:49.000701 7407 tablet_copy_client.cc:323] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.1:33705
I20251026 09:10:49.001581 6543 tablet_copy_service.cc:140] P 3ed603c643ac44ca9908b171687d7d5c: Received BeginTabletCopySession request for tablet 6459af4a4180422a8eccf12fc862499e from peer 26c251f040b9472a92191f35891e8eed ({username='slave'} at 127.0.0.1:58770)
I20251026 09:10:49.001648 6543 tablet_copy_service.cc:161] P 3ed603c643ac44ca9908b171687d7d5c: Beginning new tablet copy session on tablet 6459af4a4180422a8eccf12fc862499e from peer 26c251f040b9472a92191f35891e8eed at {username='slave'} at 127.0.0.1:58770: session id = 26c251f040b9472a92191f35891e8eed-6459af4a4180422a8eccf12fc862499e
I20251026 09:10:49.001819 7410 ts_tablet_manager.cc:933] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: Initiating tablet copy from peer 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:49.002009 7410 tablet_copy_client.cc:323] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:39689
I20251026 09:10:49.002223 6694 tablet_copy_service.cc:140] P 6d20fbac1f1243a0b383c9383a1d0a70: Received BeginTabletCopySession request for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 from peer 485b30c027fe4845b029de91594146fd ({username='slave'} at 127.0.0.1:33682)
I20251026 09:10:49.002266 6694 tablet_copy_service.cc:161] P 6d20fbac1f1243a0b383c9383a1d0a70: Beginning new tablet copy session on tablet 4bb5819b36e244d4b0ab2c0e110e51e9 from peer 485b30c027fe4845b029de91594146fd at {username='slave'} at 127.0.0.1:33682: session id = 485b30c027fe4845b029de91594146fd-4bb5819b36e244d4b0ab2c0e110e51e9
I20251026 09:10:49.002425 6543 tablet_copy_source_session.cc:215] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:49.002601 7409 ts_tablet_manager.cc:933] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: Initiating tablet copy from peer cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:49.002764 7407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6459af4a4180422a8eccf12fc862499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:49.002823 7409 tablet_copy_client.cc:323] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42631
I20251026 09:10:49.002929 6694 tablet_copy_source_session.cc:215] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:49.003176 7410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb5819b36e244d4b0ab2c0e110e51e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:49.004302 7407 tablet_copy_client.cc:806] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:49.004441 7407 tablet_copy_client.cc:670] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:49.004642 7410 tablet_copy_client.cc:806] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:49.004765 7410 tablet_copy_client.cc:670] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:49.005345 7408 tablet_copy_client.cc:1025] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:49.005381 7411 tablet_copy_client.cc:1025] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:49.007812 6618 tablet_copy_service.cc:140] P cfcab08da3db40e1974902b787ef6361: Received BeginTabletCopySession request for tablet f9723f81063e4f23bb2300f2c7bd50eb from peer 485b30c027fe4845b029de91594146fd ({username='slave'} at 127.0.0.1:44520)
I20251026 09:10:49.007889 6618 tablet_copy_service.cc:161] P cfcab08da3db40e1974902b787ef6361: Beginning new tablet copy session on tablet f9723f81063e4f23bb2300f2c7bd50eb from peer 485b30c027fe4845b029de91594146fd at {username='slave'} at 127.0.0.1:44520: session id = 485b30c027fe4845b029de91594146fd-f9723f81063e4f23bb2300f2c7bd50eb
I20251026 09:10:49.008348 6618 tablet_copy_source_session.cc:215] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:49.008736 7409 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9723f81063e4f23bb2300f2c7bd50eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:49.010282 7409 tablet_copy_client.cc:806] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:49.010407 7409 tablet_copy_client.cc:670] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:49.011039 7412 tablet_copy_client.cc:1025] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:49.039814 7381 raft_consensus.cc:993] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:49.039932 7381 raft_consensus.cc:1081] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:49.040095 6532 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44252
I20251026 09:10:49.040205 6532 raft_consensus.cc:493] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:49.040236 6532 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:49.041062 6532 raft_consensus.cc:515] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:49.041232 6532 leader_election.cc:290] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 4 election: Requested vote from peers 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689), 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:10:49.041373 7088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: true dest_uuid: "485b30c027fe4845b029de91594146fd"
I20251026 09:10:49.041448 7088 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:49.041513 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:49.041620 6681 raft_consensus.cc:3055] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 09:10:49.041649 6681 raft_consensus.cc:740] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:49.041843 6681 consensus_queue.cc:260] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:49.041999 6681 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:49.042513 7088 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 4.
I20251026 09:10:49.042680 6499 leader_election.cc:304] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 485b30c027fe4845b029de91594146fd; no voters:
I20251026 09:10:49.042814 6681 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 4.
I20251026 09:10:49.042861 7400 raft_consensus.cc:2804] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Leader election won for term 4
I20251026 09:10:49.043032 7400 raft_consensus.cc:697] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:49.043123 7400 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:49.043664 6430 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 3 to 4, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 4 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:49.045042 6532 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:49.045126 6532 raft_consensus.cc:606] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Received request to transfer leadership
I20251026 09:10:49.046541 6532 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:49.046605 6532 raft_consensus.cc:606] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Received request to transfer leadership
I20251026 09:10:49.047034 6532 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:49.047081 6532 raft_consensus.cc:606] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Received request to transfer leadership
W20251026 09:10:49.047238 6478 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 75fb5eb7df3b4c87bfdaa0b3f0efe601 already in progress
W20251026 09:10:49.047286 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 75fb5eb7df3b4c87bfdaa0b3f0efe601 already in progress
I20251026 09:10:49.072337 7400 raft_consensus.cc:993] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower 485b30c027fe4845b029de91594146fd to start an election
I20251026 09:10:49.072486 7378 raft_consensus.cc:1081] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Signalling peer 485b30c027fe4845b029de91594146fd to start an election
I20251026 09:10:49.072697 7088 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90"
dest_uuid: "485b30c027fe4845b029de91594146fd"
from {username='slave'} at 127.0.0.1:40416
I20251026 09:10:49.072779 7088 raft_consensus.cc:493] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:49.072815 7088 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:49.074129 7088 raft_consensus.cc:515] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:49.074350 7088 leader_election.cc:290] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [CANDIDATE]: Term 3 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:49.074529 6532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "485b30c027fe4845b029de91594146fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 40 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:49.074640 6532 raft_consensus.cc:3055] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:10:49.074678 6532 raft_consensus.cc:740] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:49.074741 6532 consensus_queue.cc:260] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 40, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:49.074841 6532 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:49.075492 6532 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485b30c027fe4845b029de91594146fd in term 3.
I20251026 09:10:49.075685 7052 leader_election.cc:304] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 485b30c027fe4845b029de91594146fd; no voters:
I20251026 09:10:49.075811 7404 raft_consensus.cc:2804] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:10:49.075810 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "485b30c027fe4845b029de91594146fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 40 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:49.075927 6604 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:49.076045 7404 raft_consensus.cc:697] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 3 LEADER]: Becoming Leader. State: Replica: 485b30c027fe4845b029de91594146fd, State: Running, Role: LEADER
I20251026 09:10:49.076108 7404 consensus_queue.cc:237] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 40, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:49.076711 6430 catalog_manager.cc:5649] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd reported cstate change: term changed from 2 to 3, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to 485b30c027fe4845b029de91594146fd (127.3.14.6). New cstate: current_term: 3 leader_uuid: "485b30c027fe4845b029de91594146fd" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:49.076781 6604 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485b30c027fe4845b029de91594146fd in term 3.
I20251026 09:10:49.114495 7418 ts_tablet_manager.cc:933] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: Initiating tablet copy from peer cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:49.114763 7418 tablet_copy_client.cc:323] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42631
I20251026 09:10:49.114957 6618 tablet_copy_service.cc:140] P cfcab08da3db40e1974902b787ef6361: Received BeginTabletCopySession request for tablet 22a8d92b27b14f89a4728a8f50db932b from peer 26c251f040b9472a92191f35891e8eed ({username='slave'} at 127.0.0.1:44500)
I20251026 09:10:49.115015 6618 tablet_copy_service.cc:161] P cfcab08da3db40e1974902b787ef6361: Beginning new tablet copy session on tablet 22a8d92b27b14f89a4728a8f50db932b from peer 26c251f040b9472a92191f35891e8eed at {username='slave'} at 127.0.0.1:44500: session id = 26c251f040b9472a92191f35891e8eed-22a8d92b27b14f89a4728a8f50db932b
I20251026 09:10:49.115772 6618 tablet_copy_source_session.cc:215] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:49.116146 7418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22a8d92b27b14f89a4728a8f50db932b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:49.117769 7418 tablet_copy_client.cc:806] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:49.117905 7418 tablet_copy_client.cc:670] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:49.118526 7419 tablet_copy_client.cc:1025] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:49.203373 7400 raft_consensus.cc:993] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:49.203490 7378 raft_consensus.cc:1081] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:49.203768 6681 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33632
I20251026 09:10:49.203881 6681 raft_consensus.cc:493] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:49.203923 6681 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:49.204936 6681 raft_consensus.cc:515] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:49.205109 6681 leader_election.cc:290] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 5 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:49.205294 6532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 5 candidate_status { last_received { term: 4 index: 40 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:49.205295 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 5 candidate_status { last_received { term: 4 index: 40 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:49.205397 6532 raft_consensus.cc:3055] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Stepping down as leader of term 4
I20251026 09:10:49.205410 6604 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:49.206099 6604 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 5.
I20251026 09:10:49.205417 6532 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:49.206260 6532 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 4.40, Last appended by leader: 40, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:49.206261 6650 leader_election.cc:304] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:49.206379 6532 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:49.206389 7382 raft_consensus.cc:2804] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 09:10:49.206451 7382 raft_consensus.cc:697] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:49.206514 7382 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 4.40, Last appended by leader: 40, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:49.207015 6430 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 4 to 5, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 5 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:49.207057 6532 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 5.
I20251026 09:10:49.274044 7377 raft_consensus.cc:993] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:49.274159 7377 raft_consensus.cc:1081] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:49.274361 6533 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44258
I20251026 09:10:49.274461 6533 raft_consensus.cc:493] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:49.274498 6533 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:49.275496 6533 raft_consensus.cc:515] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:49.275665 6533 leader_election.cc:290] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:49.275765 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:49.275839 6604 raft_consensus.cc:3055] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:10:49.275805 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:49.275902 6604 raft_consensus.cc:740] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:49.275919 6681 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:49.275954 6604 consensus_queue.cc:260] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:49.276028 6604 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:10:49.276624 6681 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:49.276647 6604 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 2.
I20251026 09:10:49.276800 6498 leader_election.cc:304] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:49.276907 7378 raft_consensus.cc:2804] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:10:49.276950 7378 raft_consensus.cc:697] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:49.276997 7378 consensus_queue.cc:237] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:49.277544 6430 catalog_manager.cc:5649] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 1 to 2, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 2 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:49.367321 7377 raft_consensus.cc:993] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 90365b2687854499bddaa63c08c0964f to start an election
I20251026 09:10:49.367449 7377 raft_consensus.cc:1081] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Signalling peer 90365b2687854499bddaa63c08c0964f to start an election
I20251026 09:10:49.367621 7014 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0"
dest_uuid: "90365b2687854499bddaa63c08c0964f"
from {username='slave'} at 127.0.0.1:34750
I20251026 09:10:49.367717 7014 raft_consensus.cc:493] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:49.367755 7014 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:49.368724 7014 raft_consensus.cc:515] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:49.368932 7014 leader_election.cc:290] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [CANDIDATE]: Term 5 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:49.369046 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "90365b2687854499bddaa63c08c0964f" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:49.369148 6533 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:49.369841 6533 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90365b2687854499bddaa63c08c0964f in term 5.
I20251026 09:10:49.370054 6977 leader_election.cc:304] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 90365b2687854499bddaa63c08c0964f; no voters:
I20251026 09:10:49.370118 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21bd52ca927842ccab72fc70494128d0" candidate_uuid: "90365b2687854499bddaa63c08c0964f" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:49.370194 6605 raft_consensus.cc:3055] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 09:10:49.370222 6605 raft_consensus.cc:740] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:49.370247 7422 raft_consensus.cc:2804] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 5 FOLLOWER]: Leader election won for term 5
I20251026 09:10:49.370270 6605 consensus_queue.cc:260] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:49.370343 7422 raft_consensus.cc:697] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 5 LEADER]: Becoming Leader. State: Replica: 90365b2687854499bddaa63c08c0964f, State: Running, Role: LEADER
I20251026 09:10:49.370419 6605 raft_consensus.cc:3060] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:49.370409 7422 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:49.371044 6430 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f reported cstate change: term changed from 4 to 5, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 90365b2687854499bddaa63c08c0964f (127.3.14.5). New cstate: current_term: 5 leader_uuid: "90365b2687854499bddaa63c08c0964f" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:49.371203 6605 raft_consensus.cc:2468] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90365b2687854499bddaa63c08c0964f in term 5.
I20251026 09:10:49.447681 7088 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20251026 09:10:49.447942 7378 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:49.448596 7400 raft_consensus.cc:993] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower 485b30c027fe4845b029de91594146fd to start an election
I20251026 09:10:49.448673 7400 raft_consensus.cc:1081] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Signalling peer 485b30c027fe4845b029de91594146fd to start an election
W20251026 09:10:49.449285 6496 consensus_peers.cc:597] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75fb5eb7df3b4c87bfdaa0b3f0efe601. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:49.449343 6681 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20251026 09:10:49.449478 7088 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601"
dest_uuid: "485b30c027fe4845b029de91594146fd"
from {username='slave'} at 127.0.0.1:40416
I20251026 09:10:49.449548 7088 raft_consensus.cc:493] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:49.449548 7400 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:49.449626 7088 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:49.450620 7088 raft_consensus.cc:515] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:49.450937 7088 leader_election.cc:290] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [CANDIDATE]: Term 5 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:49.451126 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "485b30c027fe4845b029de91594146fd" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:49.451160 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75fb5eb7df3b4c87bfdaa0b3f0efe601" candidate_uuid: "485b30c027fe4845b029de91594146fd" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:49.451203 6684 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:49.451227 6533 raft_consensus.cc:3055] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Stepping down as leader of term 4
I20251026 09:10:49.451251 6533 raft_consensus.cc:740] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:49.451308 6533 consensus_queue.cc:260] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:49.451417 6533 raft_consensus.cc:3060] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:49.451952 6533 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485b30c027fe4845b029de91594146fd in term 5.
I20251026 09:10:49.452095 6684 raft_consensus.cc:2468] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485b30c027fe4845b029de91594146fd in term 5.
I20251026 09:10:49.452275 7054 leader_election.cc:304] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485b30c027fe4845b029de91594146fd, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:49.452379 7404 raft_consensus.cc:2804] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Leader election won for term 5
I20251026 09:10:49.452435 7404 raft_consensus.cc:697] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 5 LEADER]: Becoming Leader. State: Replica: 485b30c027fe4845b029de91594146fd, State: Running, Role: LEADER
I20251026 09:10:49.452495 7404 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:49.454843 6430 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd reported cstate change: term changed from 4 to 5, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to 485b30c027fe4845b029de91594146fd (127.3.14.6). New cstate: current_term: 5 leader_uuid: "485b30c027fe4845b029de91594146fd" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:49.523830 6605 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251026 09:10:49.524066 7404 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:49.525588 6533 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251026 09:10:49.525833 7431 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20251026 09:10:49.649371 6650 consensus_peers.cc:597] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841): Couldn't send request to peer 26c251f040b9472a92191f35891e8eed. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:49.661671 6604 raft_consensus.cc:1275] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:49.661911 7400 consensus_queue.cc:1048] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:49.663116 6684 raft_consensus.cc:1275] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 37. (index mismatch)
I20251026 09:10:49.663396 7400 consensus_queue.cc:1048] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251026 09:10:49.773463 6533 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Refusing update from remote peer 90365b2687854499bddaa63c08c0964f: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 5 index: 43. (index mismatch)
I20251026 09:10:49.773607 6604 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 4 index: 40. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
I20251026 09:10:49.773715 7423 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:49.773865 7381 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:49.774888 6533 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 4 index: 40. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
I20251026 09:10:49.775494 6604 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Refusing update from remote peer 90365b2687854499bddaa63c08c0964f: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 5 index: 43. (index mismatch)
I20251026 09:10:49.775599 7434 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:49.775763 7422 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
W20251026 09:10:49.834605 7052 consensus_peers.cc:597] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75fb5eb7df3b4c87bfdaa0b3f0efe601. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:49.892587 6681 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 5 index: 43. (index mismatch)
I20251026 09:10:49.892870 7431 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:49.893918 6532 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 5 index: 43. (index mismatch)
I20251026 09:10:49.894112 7404 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:50.381678 7445 ts_tablet_manager.cc:933] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: Initiating tablet copy from peer 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:10:50.381919 7445 tablet_copy_client.cc:323] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.6:35539
I20251026 09:10:50.384992 7099 tablet_copy_service.cc:140] P 485b30c027fe4845b029de91594146fd: Received BeginTabletCopySession request for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 from peer 90365b2687854499bddaa63c08c0964f ({username='slave'} at 127.0.0.1:43432)
I20251026 09:10:50.385085 7099 tablet_copy_service.cc:161] P 485b30c027fe4845b029de91594146fd: Beginning new tablet copy session on tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 from peer 90365b2687854499bddaa63c08c0964f at {username='slave'} at 127.0.0.1:43432: session id = 90365b2687854499bddaa63c08c0964f-75fb5eb7df3b4c87bfdaa0b3f0efe601
I20251026 09:10:50.385771 7099 tablet_copy_source_session.cc:215] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:50.386147 7445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:50.387645 7445 tablet_copy_client.cc:806] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:50.387792 7445 tablet_copy_client.cc:670] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:50.388473 7446 tablet_copy_client.cc:1025] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:50.856501 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:50.856984 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:50.857079 6684 raft_consensus.cc:606] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:50.857668 6532 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:50.857771 6532 raft_consensus.cc:606] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:10:50.857969 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 09:10:50.858038 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:50.983795 7400 raft_consensus.cc:993] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:50.983913 7400 raft_consensus.cc:1081] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:50.984148 6604 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47314
I20251026 09:10:50.984283 6604 raft_consensus.cc:493] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:50.984323 6604 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:50.985136 6604 raft_consensus.cc:515] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:50.985309 6604 leader_election.cc:290] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841)
I20251026 09:10:50.985467 6532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:50.985467 6938 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: true dest_uuid: "26c251f040b9472a92191f35891e8eed"
I20251026 09:10:50.985580 6532 raft_consensus.cc:3055] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 LEADER]: Stepping down as leader of term 3
I20251026 09:10:50.985587 6938 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:50.985608 6532 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:50.985752 6532 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:50.985839 6532 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:50.986292 6938 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 4.
I20251026 09:10:50.986438 6532 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 4.
I20251026 09:10:50.986544 6575 leader_election.cc:304] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c251f040b9472a92191f35891e8eed, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:50.986655 7429 raft_consensus.cc:2804] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 09:10:50.986791 7429 raft_consensus.cc:697] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:50.986850 7429 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:50.987358 6430 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 3 to 4, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:50.998548 7405 tablet_copy_client.cc:538] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:50.999809 7405 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: Bootstrap starting.
I20251026 09:10:51.005637 7407 tablet_copy_client.cc:538] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:51.005669 7405 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=41 overwritten=0 applied=41 ignored=0} inserts{seen=148 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:51.005964 7405 tablet_bootstrap.cc:492] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: Bootstrap complete.
I20251026 09:10:51.006156 7405 ts_tablet_manager.cc:1403] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20251026 09:10:51.006165 7410 tablet_copy_client.cc:538] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:51.006382 7405 raft_consensus.cc:359] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:51.006484 7405 raft_consensus.cc:740] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 90365b2687854499bddaa63c08c0964f, State: Initialized, Role: LEARNER
I20251026 09:10:51.006531 7405 consensus_queue.cc:260] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:51.006705 7405 ts_tablet_manager.cc:1434] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:51.006953 6694 tablet_copy_service.cc:342] P 6d20fbac1f1243a0b383c9383a1d0a70: Request end of tablet copy session 90365b2687854499bddaa63c08c0964f-e002d8e9fb004210be27d87c5a23f9c3 received from {username='slave'} at 127.0.0.1:44960
I20251026 09:10:51.007030 6694 tablet_copy_service.cc:434] P 6d20fbac1f1243a0b383c9383a1d0a70: ending tablet copy session 90365b2687854499bddaa63c08c0964f-e002d8e9fb004210be27d87c5a23f9c3 on tablet e002d8e9fb004210be27d87c5a23f9c3 with peer 90365b2687854499bddaa63c08c0964f
I20251026 09:10:51.008826 7410 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: Bootstrap starting.
I20251026 09:10:51.011276 7409 tablet_copy_client.cc:538] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:51.011734 7407 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: Bootstrap starting.
I20251026 09:10:51.012446 7409 tablet_bootstrap.cc:492] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: Bootstrap starting.
I20251026 09:10:51.016383 7410 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=41 overwritten=0 applied=41 ignored=0} inserts{seen=140 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:51.016746 7410 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: Bootstrap complete.
I20251026 09:10:51.016892 7410 ts_tablet_manager.cc:1403] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20251026 09:10:51.016999 7407 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=145 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:51.017235 7410 raft_consensus.cc:359] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:51.017367 7410 raft_consensus.cc:740] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 485b30c027fe4845b029de91594146fd, State: Initialized, Role: LEARNER
I20251026 09:10:51.017432 7410 consensus_queue.cc:260] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:51.017566 7407 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: Bootstrap complete.
I20251026 09:10:51.017676 7410 ts_tablet_manager.cc:1434] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:10:51.017959 7407 ts_tablet_manager.cc:1403] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20251026 09:10:51.018388 6694 tablet_copy_service.cc:342] P 6d20fbac1f1243a0b383c9383a1d0a70: Request end of tablet copy session 485b30c027fe4845b029de91594146fd-4bb5819b36e244d4b0ab2c0e110e51e9 received from {username='slave'} at 127.0.0.1:33682
I20251026 09:10:51.018445 6694 tablet_copy_service.cc:434] P 6d20fbac1f1243a0b383c9383a1d0a70: ending tablet copy session 485b30c027fe4845b029de91594146fd-4bb5819b36e244d4b0ab2c0e110e51e9 on tablet 4bb5819b36e244d4b0ab2c0e110e51e9 with peer 485b30c027fe4845b029de91594146fd
I20251026 09:10:51.018708 7407 raft_consensus.cc:359] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:51.018826 7407 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Initialized, Role: LEARNER
I20251026 09:10:51.018870 7407 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 4.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:51.018909 7409 tablet_bootstrap.cc:492] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=145 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:51.019083 7407 ts_tablet_manager.cc:1434] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:10:51.019107 7409 tablet_bootstrap.cc:492] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: Bootstrap complete.
I20251026 09:10:51.019281 6543 tablet_copy_service.cc:342] P 3ed603c643ac44ca9908b171687d7d5c: Request end of tablet copy session 26c251f040b9472a92191f35891e8eed-6459af4a4180422a8eccf12fc862499e received from {username='slave'} at 127.0.0.1:58770
I20251026 09:10:51.019351 6543 tablet_copy_service.cc:434] P 3ed603c643ac44ca9908b171687d7d5c: ending tablet copy session 26c251f040b9472a92191f35891e8eed-6459af4a4180422a8eccf12fc862499e on tablet 6459af4a4180422a8eccf12fc862499e with peer 26c251f040b9472a92191f35891e8eed
I20251026 09:10:51.019510 7409 ts_tablet_manager.cc:1403] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20251026 09:10:51.019687 7409 raft_consensus.cc:359] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:51.019764 7409 raft_consensus.cc:740] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 485b30c027fe4845b029de91594146fd, State: Initialized, Role: LEARNER
I20251026 09:10:51.019797 7409 consensus_queue.cc:260] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 2.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:51.019960 7409 ts_tablet_manager.cc:1434] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:51.020153 6618 tablet_copy_service.cc:342] P cfcab08da3db40e1974902b787ef6361: Request end of tablet copy session 485b30c027fe4845b029de91594146fd-f9723f81063e4f23bb2300f2c7bd50eb received from {username='slave'} at 127.0.0.1:44520
I20251026 09:10:51.020215 6618 tablet_copy_service.cc:434] P cfcab08da3db40e1974902b787ef6361: ending tablet copy session 485b30c027fe4845b029de91594146fd-f9723f81063e4f23bb2300f2c7bd50eb on tablet f9723f81063e4f23bb2300f2c7bd50eb with peer 485b30c027fe4845b029de91594146fd
I20251026 09:10:51.033565 7014 raft_consensus.cc:1217] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [term 3 LEARNER]: Deduplicated request from leader. Original: 3.40->[3.41-3.41] Dedup: 3.41->[]
I20251026 09:10:51.090171 7088 raft_consensus.cc:1217] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.37->[2.38-2.38] Dedup: 2.38->[]
I20251026 09:10:51.118790 7418 tablet_copy_client.cc:538] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:51.119724 7418 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: Bootstrap starting.
I20251026 09:10:51.123486 7418 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=149 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:51.123713 7418 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: Bootstrap complete.
I20251026 09:10:51.123842 7418 ts_tablet_manager.cc:1403] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20251026 09:10:51.124018 7418 raft_consensus.cc:359] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:51.124089 7418 raft_consensus.cc:740] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Initialized, Role: LEARNER
I20251026 09:10:51.124123 7418 consensus_queue.cc:260] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 2.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:51.124233 7418 ts_tablet_manager.cc:1434] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:51.124430 6618 tablet_copy_service.cc:342] P cfcab08da3db40e1974902b787ef6361: Request end of tablet copy session 26c251f040b9472a92191f35891e8eed-22a8d92b27b14f89a4728a8f50db932b received from {username='slave'} at 127.0.0.1:44500
I20251026 09:10:51.124496 6618 tablet_copy_service.cc:434] P cfcab08da3db40e1974902b787ef6361: ending tablet copy session 26c251f040b9472a92191f35891e8eed-22a8d92b27b14f89a4728a8f50db932b on tablet 22a8d92b27b14f89a4728a8f50db932b with peer 26c251f040b9472a92191f35891e8eed
I20251026 09:10:51.270370 7381 raft_consensus.cc:993] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:51.270476 7381 raft_consensus.cc:1081] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:51.270630 6532 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44252
I20251026 09:10:51.270738 6532 raft_consensus.cc:493] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:51.270777 6532 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Advancing to term 6
I20251026 09:10:51.271610 6532 raft_consensus.cc:515] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:51.271831 6532 leader_election.cc:290] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 6 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:51.271935 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 6 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:51.271994 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 6 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:51.272150 6604 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Advancing to term 6
I20251026 09:10:51.272167 6684 raft_consensus.cc:3055] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 09:10:51.272284 6684 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:51.272359 6684 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 5.41, Last appended by leader: 41, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:51.272459 6684 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Advancing to term 6
I20251026 09:10:51.273053 6604 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 6.
I20251026 09:10:51.273128 6684 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 6.
I20251026 09:10:51.273250 6499 leader_election.cc:304] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:51.273348 7400 raft_consensus.cc:2804] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 FOLLOWER]: Leader election won for term 6
I20251026 09:10:51.273404 7400 raft_consensus.cc:697] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:51.273465 7400 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 5.41, Last appended by leader: 41, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:51.274140 6430 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 5 to 6, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 6 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:51.407177 7088 raft_consensus.cc:1217] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 LEARNER]: Deduplicated request from leader. Original: 3.40->[3.41-3.41] Dedup: 3.41->[]
I20251026 09:10:51.414696 6532 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 4 index: 41. (index mismatch)
I20251026 09:10:51.414949 7429 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:51.416145 6938 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 4 index: 41. (index mismatch)
I20251026 09:10:51.416389 7467 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:51.499950 6938 raft_consensus.cc:1217] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 LEARNER]: Deduplicated request from leader. Original: 2.37->[2.38-2.38] Dedup: 2.38->[]
I20251026 09:10:51.529930 7471 raft_consensus.cc:1064] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: attempting to promote NON_VOTER 90365b2687854499bddaa63c08c0964f to VOTER
I20251026 09:10:51.530247 7471 consensus_queue.cc:237] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 36, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:51.530825 7014 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [term 3 LEARNER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 3 index: 42. (index mismatch)
I20251026 09:10:51.530825 6938 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 3 index: 42. (index mismatch)
I20251026 09:10:51.531034 7471 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:51.531133 6605 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 3 index: 42. (index mismatch)
I20251026 09:10:51.531183 7381 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:51.531519 7381 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:51.532258 7471 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.42: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:51.532366 6605 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Committing config change with OpId 3.42: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:51.532624 7014 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Committing config change with OpId 3.42: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:51.533692 6430 catalog_manager.cc:5649] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f reported cstate change: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:51.533989 6938 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Committing config change with OpId 3.42: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:51.535859 6681 consensus_queue.cc:237] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 3.42, Last appended by leader: 36, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:51.536273 7014 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 42. Preceding OpId from leader: term: 3 index: 43. (index mismatch)
I20251026 09:10:51.536441 6938 raft_consensus.cc:1275] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 42. Preceding OpId from leader: term: 3 index: 43. (index mismatch)
I20251026 09:10:51.536481 7381 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:51.536744 7381 consensus_queue.cc:1048] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:51.537140 7381 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.43: config changed from index 42 to 43, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:51.537447 7014 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Committing config change with OpId 3.43: config changed from index 42 to 43, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:51.537411 6938 raft_consensus.cc:2955] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Committing config change with OpId 3.43: config changed from index 42 to 43, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:51.538507 6428 catalog_manager.cc:5649] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 42 to 43, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:51.538892 6416 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet e002d8e9fb004210be27d87c5a23f9c3 with cas_config_opid_index 42: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:51.539453 6596 tablet_service.cc:1552] Processing DeleteTablet for tablet e002d8e9fb004210be27d87c5a23f9c3 with delete_type TABLET_DATA_TOMBSTONED (TS cfcab08da3db40e1974902b787ef6361 not found in new config with opid_index 43) from {username='slave'} at 127.0.0.1:47308
I20251026 09:10:51.539701 7477 tablet_replica.cc:333] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:10:51.539776 7477 raft_consensus.cc:2243] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:51.539880 7477 raft_consensus.cc:2272] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:51.540211 7477 ts_tablet_manager.cc:1916] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:51.542093 7477 ts_tablet_manager.cc:1929] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.42
I20251026 09:10:51.542163 7477 log.cc:1199] T e002d8e9fb004210be27d87c5a23f9c3 P cfcab08da3db40e1974902b787ef6361: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/e002d8e9fb004210be27d87c5a23f9c3
I20251026 09:10:51.542598 6417 catalog_manager.cc:4985] TS cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631): tablet e002d8e9fb004210be27d87c5a23f9c3 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:51.698496 7429 raft_consensus.cc:1064] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: attempting to promote NON_VOTER 485b30c027fe4845b029de91594146fd to VOTER
I20251026 09:10:51.698767 7429 consensus_queue.cc:237] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 2.38, Last appended by leader: 36, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:51.699419 7088 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 2 LEARNER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20251026 09:10:51.699594 6681 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20251026 09:10:51.699623 7452 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:51.699702 6533 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20251026 09:10:51.699862 7452 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:51.699955 7467 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:51.700662 7429 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.701021 6533 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.701874 7088 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.702057 6684 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.702265 6429 catalog_manager.cc:5649] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 38 to 39, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.703303 6604 consensus_queue.cc:237] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:51.703920 6684 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20251026 09:10:51.703930 7088 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20251026 09:10:51.704150 7479 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:51.704406 7452 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:51.704735 7479 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.704833 7088 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.705076 6684 raft_consensus.cc:2955] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.705906 6429 catalog_manager.cc:5649] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd reported cstate change: config changed from index 39 to 40, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.706172 6417 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet f9723f81063e4f23bb2300f2c7bd50eb with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:51.706388 6684 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 5 index: 41. Preceding OpId from leader: term: 6 index: 42. (index mismatch)
I20251026 09:10:51.706653 7451 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:51.706760 6518 tablet_service.cc:1552] Processing DeleteTablet for tablet f9723f81063e4f23bb2300f2c7bd50eb with delete_type TABLET_DATA_TOMBSTONED (TS 3ed603c643ac44ca9908b171687d7d5c not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:51.707032 7483 tablet_replica.cc:333] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:10:51.707104 7483 raft_consensus.cc:2243] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:51.707172 7483 raft_consensus.cc:2272] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:51.707648 7483 ts_tablet_manager.cc:1916] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:51.708070 6938 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 4 LEARNER]: Advancing to term 6
I20251026 09:10:51.708810 6938 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 LEARNER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 4 index: 40. Preceding OpId from leader: term: 6 index: 42. (index mismatch)
I20251026 09:10:51.709157 6605 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 5 index: 41. Preceding OpId from leader: term: 6 index: 42. (index mismatch)
I20251026 09:10:51.709344 7400 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:51.709468 7400 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:51.709540 7483 ts_tablet_manager.cc:1929] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.39
I20251026 09:10:51.709594 7483 log.cc:1199] T f9723f81063e4f23bb2300f2c7bd50eb P 3ed603c643ac44ca9908b171687d7d5c: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/f9723f81063e4f23bb2300f2c7bd50eb
I20251026 09:10:51.709949 6414 catalog_manager.cc:4985] TS 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705): tablet f9723f81063e4f23bb2300f2c7bd50eb (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:51.710668 7451 raft_consensus.cc:1064] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: attempting to promote NON_VOTER 26c251f040b9472a92191f35891e8eed to VOTER
I20251026 09:10:51.710867 7451 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 6.42, Last appended by leader: 41, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:51.711284 6681 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 42. Preceding OpId from leader: term: 6 index: 43. (index mismatch)
I20251026 09:10:51.711418 6938 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 LEARNER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 42. Preceding OpId from leader: term: 6 index: 43. (index mismatch)
I20251026 09:10:51.711632 6605 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 42. Preceding OpId from leader: term: 6 index: 43. (index mismatch)
I20251026 09:10:51.711612 7490 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:51.711784 7451 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:51.711923 7485 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:51.712634 7485 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Committing config change with OpId 6.43: config changed from index 40 to 43, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:51.712766 6681 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Committing config change with OpId 6.43: config changed from index 40 to 43, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:51.712956 6605 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Committing config change with OpId 6.43: config changed from index 40 to 43, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:51.713762 6938 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Committing config change with OpId 6.43: config changed from index 40 to 43, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:51.713981 6428 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 40 to 43, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:51.715148 6533 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 6.43, Last appended by leader: 41, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:51.715610 6681 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 43. Preceding OpId from leader: term: 6 index: 44. (index mismatch)
I20251026 09:10:51.715624 6938 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 43. Preceding OpId from leader: term: 6 index: 44. (index mismatch)
I20251026 09:10:51.715799 7490 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:51.715905 7451 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:51.716487 7400 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Committing config change with OpId 6.44: config changed from index 43 to 44, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:51.716574 6938 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Committing config change with OpId 6.44: config changed from index 43 to 44, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:51.717558 6414 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 6459af4a4180422a8eccf12fc862499e with cas_config_opid_index 43: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:51.717731 6428 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed reported cstate change: config changed from index 43 to 44, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New cstate: current_term: 6 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:51.718007 6681 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Committing config change with OpId 6.44: config changed from index 43 to 44, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:51.718488 6596 tablet_service.cc:1552] Processing DeleteTablet for tablet 6459af4a4180422a8eccf12fc862499e with delete_type TABLET_DATA_TOMBSTONED (TS cfcab08da3db40e1974902b787ef6361 not found in new config with opid_index 44) from {username='slave'} at 127.0.0.1:47308
I20251026 09:10:51.718595 7477 tablet_replica.cc:333] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:10:51.718659 7477 raft_consensus.cc:2243] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:51.718760 7477 raft_consensus.cc:2272] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:51.719022 7477 ts_tablet_manager.cc:1916] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:51.720609 7477 ts_tablet_manager.cc:1929] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.43
I20251026 09:10:51.720678 7477 log.cc:1199] T 6459af4a4180422a8eccf12fc862499e P cfcab08da3db40e1974902b787ef6361: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/6459af4a4180422a8eccf12fc862499e
I20251026 09:10:51.721057 6417 catalog_manager.cc:4985] TS cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631): tablet 6459af4a4180422a8eccf12fc862499e (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:51.852885 7471 raft_consensus.cc:1064] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: attempting to promote NON_VOTER 485b30c027fe4845b029de91594146fd to VOTER
I20251026 09:10:51.853149 7471 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 37, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:51.853691 7088 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 LEARNER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 3 index: 42. (index mismatch)
I20251026 09:10:51.853708 7014 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 3 index: 42. (index mismatch)
I20251026 09:10:51.853853 6532 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 3 index: 42. (index mismatch)
I20251026 09:10:51.853899 7381 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:51.854001 7450 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:51.854077 7473 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:51.855019 7381 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.42: config changed from index 41 to 42, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.856181 7088 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Committing config change with OpId 3.42: config changed from index 41 to 42, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.856238 6532 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Committing config change with OpId 3.42: config changed from index 41 to 42, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.856328 6430 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 41 to 42, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:51.857131 7014 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Committing config change with OpId 3.42: config changed from index 41 to 42, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.858443 6681 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 3.42, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:51.858865 7088 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 42. Preceding OpId from leader: term: 3 index: 43. (index mismatch)
I20251026 09:10:51.859076 7014 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 42. Preceding OpId from leader: term: 3 index: 43. (index mismatch)
I20251026 09:10:51.859045 7381 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:51.859258 7450 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:51.859680 7471 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.43: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.859789 7088 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Committing config change with OpId 3.43: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.859937 7014 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Committing config change with OpId 3.43: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.860814 6428 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd reported cstate change: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:51.861161 6416 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 with cas_config_opid_index 42: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:51.861670 6518 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 with delete_type TABLET_DATA_TOMBSTONED (TS 3ed603c643ac44ca9908b171687d7d5c not found in new config with opid_index 43) from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:51.861793 7483 tablet_replica.cc:333] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:10:51.861862 7483 raft_consensus.cc:2243] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:51.862062 7483 raft_consensus.cc:2272] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:51.862342 7483 ts_tablet_manager.cc:1916] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:51.864223 7483 ts_tablet_manager.cc:1929] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.42
I20251026 09:10:51.864311 7483 log.cc:1199] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 3ed603c643ac44ca9908b171687d7d5c: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/4bb5819b36e244d4b0ab2c0e110e51e9
I20251026 09:10:51.864732 6414 catalog_manager.cc:4985] TS 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705): tablet 4bb5819b36e244d4b0ab2c0e110e51e9 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:52.077266 7452 raft_consensus.cc:1064] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: attempting to promote NON_VOTER 26c251f040b9472a92191f35891e8eed to VOTER
I20251026 09:10:52.077534 7452 consensus_queue.cc:237] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 2.38, Last appended by leader: 36, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:52.078150 6532 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20251026 09:10:52.078198 6938 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 LEARNER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20251026 09:10:52.078145 6684 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20251026 09:10:52.078490 7467 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:52.078586 7452 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:52.078701 7479 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:52.079874 7452 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:52.080307 6938 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:52.081091 6532 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:52.081281 6684 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index 38 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:52.081360 6430 catalog_manager.cc:5649] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed reported cstate change: config changed from index 38 to 39, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:52.083148 6604 consensus_queue.cc:237] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:10:52.083604 6938 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20251026 09:10:52.083678 6532 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20251026 09:10:52.083820 7429 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:52.083917 7467 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:52.084649 7429 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:52.084777 6532 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:52.085636 6417 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 22a8d92b27b14f89a4728a8f50db932b with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:52.085703 6427 catalog_manager.cc:5649] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:52.085960 6938 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:10:52.086617 6665 tablet_service.cc:1552] Processing DeleteTablet for tablet 22a8d92b27b14f89a4728a8f50db932b with delete_type TABLET_DATA_TOMBSTONED (TS 6d20fbac1f1243a0b383c9383a1d0a70 not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:33620
I20251026 09:10:52.086911 7499 tablet_replica.cc:333] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:10:52.087034 7499 raft_consensus.cc:2243] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:52.087186 7499 raft_consensus.cc:2272] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:52.087536 7499 ts_tablet_manager.cc:1916] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:52.088985 7499 ts_tablet_manager.cc:1929] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.39
I20251026 09:10:52.089043 7499 log.cc:1199] T 22a8d92b27b14f89a4728a8f50db932b P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/22a8d92b27b14f89a4728a8f50db932b
I20251026 09:10:52.089442 6416 catalog_manager.cc:4985] TS 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689): tablet 22a8d92b27b14f89a4728a8f50db932b (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:52.388842 7445 tablet_copy_client.cc:538] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:52.389846 7445 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: Bootstrap starting.
I20251026 09:10:52.394418 7445 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=43 ignored=0} inserts{seen=147 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:52.394672 7445 tablet_bootstrap.cc:492] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: Bootstrap complete.
I20251026 09:10:52.394814 7445 ts_tablet_manager.cc:1403] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251026 09:10:52.395004 7445 raft_consensus.cc:359] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:52.395081 7445 raft_consensus.cc:740] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 90365b2687854499bddaa63c08c0964f, State: Initialized, Role: LEARNER
I20251026 09:10:52.395110 7445 consensus_queue.cc:260] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 5.43, Last appended by leader: 43, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:52.395221 7445 ts_tablet_manager.cc:1434] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:52.395550 7099 tablet_copy_service.cc:342] P 485b30c027fe4845b029de91594146fd: Request end of tablet copy session 90365b2687854499bddaa63c08c0964f-75fb5eb7df3b4c87bfdaa0b3f0efe601 received from {username='slave'} at 127.0.0.1:43432
I20251026 09:10:52.395625 7099 tablet_copy_service.cc:434] P 485b30c027fe4845b029de91594146fd: ending tablet copy session 90365b2687854499bddaa63c08c0964f-75fb5eb7df3b4c87bfdaa0b3f0efe601 on tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 with peer 90365b2687854499bddaa63c08c0964f
I20251026 09:10:52.811636 7014 raft_consensus.cc:1217] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [term 5 LEARNER]: Deduplicated request from leader. Original: 4.42->[5.43-5.43] Dedup: 5.43->[]
I20251026 09:10:52.858278 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:52.858994 6532 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:52.859112 6532 raft_consensus.cc:606] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Received request to transfer leadership to TS 6d20fbac1f1243a0b383c9383a1d0a70
I20251026 09:10:52.859618 6605 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:10:52.859714 6605 raft_consensus.cc:606] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Received request to transfer leadership to TS 6d20fbac1f1243a0b383c9383a1d0a70
I20251026 09:10:52.859939 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 09:10:52.860010 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:53.086851 6532 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 6.44, Last appended by leader: 41, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
W20251026 09:10:53.087697 6496 consensus_peers.cc:597] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6459af4a4180422a8eccf12fc862499e. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:53.087810 6684 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 44. Preceding OpId from leader: term: 6 index: 45. (index mismatch)
I20251026 09:10:53.087895 6938 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 44. Preceding OpId from leader: term: 6 index: 45. (index mismatch)
I20251026 09:10:53.088029 7502 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:10:53.088155 7503 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:10:53.088886 7490 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Committing config change with OpId 6.45: config changed from index 44 to 45, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:53.089010 6938 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Committing config change with OpId 6.45: config changed from index 44 to 45, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:53.089074 6684 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Committing config change with OpId 6.45: config changed from index 44 to 45, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:53.090255 6428 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 44 to 45, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New cstate: current_term: 6 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:53.090317 6605 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 4.41, Last appended by leader: 40, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:53.090833 6938 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20251026 09:10:53.090857 6532 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
W20251026 09:10:53.090898 6575 consensus_peers.cc:597] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 -> Peer 485b30c027fe4845b029de91594146fd (127.3.14.6:35539): Couldn't send request to peer 485b30c027fe4845b029de91594146fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff5ceda9af754029a9b5ac636a17ea70. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:53.091017 7508 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:53.091105 7429 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:53.091955 7508 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Committing config change with OpId 4.42: config changed from index 40 to 42, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:53.092126 6532 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Committing config change with OpId 4.42: config changed from index 40 to 42, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:53.093318 6427 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 40 to 42, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:53.093381 6605 consensus_queue.cc:237] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:53.093907 6938 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Committing config change with OpId 4.42: config changed from index 40 to 42, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:53.094746 6532 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch)
W20251026 09:10:53.094784 6572 consensus_peers.cc:597] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 -> Peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): Couldn't send request to peer 90365b2687854499bddaa63c08c0964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22a8d92b27b14f89a4728a8f50db932b. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:53.094929 7508 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:53.094967 6938 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch)
I20251026 09:10:53.095172 7508 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:53.095773 7429 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.41: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:53.096083 6532 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.41: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
I20251026 09:10:53.097141 6428 catalog_manager.cc:5649] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 09:10:53.097312 6478 auto_rebalancer.cc:253] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20251026 09:10:53.097501 6938 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Committing config change with OpId 2.41: config changed from index 40 to 41, NON_VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } } }
W20251026 09:10:53.098393 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 90072194423a4007b85cbc3f31889400, TS cfcab08da3db40e1974902b787ef6361 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:53.098438 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 90072194423a4007b85cbc3f31889400, TS cfcab08da3db40e1974902b787ef6361 -> TS 26c251f040b9472a92191f35891e8eed move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
W20251026 09:10:53.099684 6478 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ad759f01ba6c4154968854b4b9378651, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
W20251026 09:10:53.099731 6478 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet ad759f01ba6c4154968854b4b9378651, TS 6d20fbac1f1243a0b383c9383a1d0a70 -> TS 485b30c027fe4845b029de91594146fd move failed, destination replica disappeared from tablet's Raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:53.171255 7514 raft_consensus.cc:993] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:53.171419 7514 raft_consensus.cc:1081] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:53.171661 6684 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33638
I20251026 09:10:53.171782 6684 raft_consensus.cc:493] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:53.171819 6684 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:53.172742 6684 raft_consensus.cc:515] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:53.172928 6684 leader_election.cc:290] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:10:53.173089 7088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 40 } } ignore_live_leader: true dest_uuid: "485b30c027fe4845b029de91594146fd"
I20251026 09:10:53.173185 7088 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:53.174050 7088 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:53.174223 6650 leader_election.cc:304] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485b30c027fe4845b029de91594146fd, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:53.174316 7381 raft_consensus.cc:2804] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:10:53.174489 7381 raft_consensus.cc:697] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:53.174566 7381 consensus_queue.cc:237] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 40, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:53.175156 6427 catalog_manager.cc:5649] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 2 to 3, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:53.173084 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 40 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:53.175472 6605 raft_consensus.cc:3055] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:10:53.175518 6605 raft_consensus.cc:740] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:53.175592 6605 consensus_queue.cc:260] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 40, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:53.175679 6605 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:53.176463 6605 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:53.259169 7416 raft_consensus.cc:1064] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: attempting to promote NON_VOTER 90365b2687854499bddaa63c08c0964f to VOTER
I20251026 09:10:53.259483 7416 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 5.43, Last appended by leader: 42, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:53.260154 6532 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 5 index: 44. (index mismatch)
I20251026 09:10:53.260147 7014 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [term 5 LEARNER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 5 index: 44. (index mismatch)
I20251026 09:10:53.260147 6684 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 5 index: 44. (index mismatch)
I20251026 09:10:53.260454 7520 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:53.260655 7523 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:53.260818 7522 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:53.262238 7523 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 5 LEADER]: Committing config change with OpId 5.44: config changed from index 41 to 44, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:53.262374 6684 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Committing config change with OpId 5.44: config changed from index 41 to 44, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:53.262482 6532 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Committing config change with OpId 5.44: config changed from index 41 to 44, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:53.264004 7014 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [term 5 FOLLOWER]: Committing config change with OpId 5.44: config changed from index 41 to 44, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:53.264940 6427 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd reported cstate change: config changed from index 41 to 44, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "485b30c027fe4845b029de91594146fd" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:53.269379 7088 consensus_queue.cc:237] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 5.44, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:53.269836 7014 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [term 5 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 5 index: 44. Preceding OpId from leader: term: 5 index: 45. (index mismatch)
I20251026 09:10:53.269850 6684 raft_consensus.cc:1275] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 5 index: 44. Preceding OpId from leader: term: 5 index: 45. (index mismatch)
I20251026 09:10:53.270107 7520 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:10:53.270273 7523 consensus_queue.cc:1048] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:10:53.270746 7520 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 5 LEADER]: Committing config change with OpId 5.45: config changed from index 44 to 45, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:53.270900 7014 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [term 5 FOLLOWER]: Committing config change with OpId 5.45: config changed from index 44 to 45, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:53.270931 6684 raft_consensus.cc:2955] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Committing config change with OpId 5.45: config changed from index 44 to 45, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:53.272053 6417 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 with cas_config_opid_index 44: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:53.272117 6430 catalog_manager.cc:5649] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f reported cstate change: config changed from index 44 to 45, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New cstate: current_term: 5 leader_uuid: "485b30c027fe4845b029de91594146fd" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:53.272985 6518 tablet_service.cc:1552] Processing DeleteTablet for tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 with delete_type TABLET_DATA_TOMBSTONED (TS 3ed603c643ac44ca9908b171687d7d5c not found in new config with opid_index 45) from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:53.273257 7529 tablet_replica.cc:333] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:10:53.273422 7529 raft_consensus.cc:2243] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:53.273559 7529 raft_consensus.cc:2272] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:53.273849 7529 ts_tablet_manager.cc:1916] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:53.275360 7529 ts_tablet_manager.cc:1929] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.44
I20251026 09:10:53.275429 7529 log.cc:1199] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 3ed603c643ac44ca9908b171687d7d5c: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/75fb5eb7df3b4c87bfdaa0b3f0efe601
I20251026 09:10:53.275838 6414 catalog_manager.cc:4985] TS 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705): tablet 75fb5eb7df3b4c87bfdaa0b3f0efe601 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:53.354408 7503 raft_consensus.cc:993] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:53.354534 7503 raft_consensus.cc:1081] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:53.354703 6684 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33632
I20251026 09:10:53.354789 6684 raft_consensus.cc:493] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:53.354835 6684 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:53.356029 6684 raft_consensus.cc:515] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:53.356215 6684 leader_election.cc:290] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:53.356323 6532 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:53.356418 6532 raft_consensus.cc:3055] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:10:53.356496 6532 raft_consensus.cc:740] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:53.356417 6604 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:53.356527 6604 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:53.356554 6532 consensus_queue.cc:260] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:53.356632 6532 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:10:53.357328 6604 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:53.357328 6532 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:10:53.357633 6650 leader_election.cc:304] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:53.357750 7381 raft_consensus.cc:2804] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:10:53.357806 7381 raft_consensus.cc:697] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:53.357863 7381 consensus_queue.cc:237] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:53.358480 6427 catalog_manager.cc:5649] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 2 to 3, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } }
I20251026 09:10:53.562988 6604 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251026 09:10:53.563263 7518 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:53.564698 7088 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251026 09:10:53.565003 7381 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:53.619014 7536 ts_tablet_manager.cc:933] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: Initiating tablet copy from peer 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705)
I20251026 09:10:53.619204 7536 tablet_copy_client.cc:323] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.1:33705
I20251026 09:10:53.619431 6543 tablet_copy_service.cc:140] P 3ed603c643ac44ca9908b171687d7d5c: Received BeginTabletCopySession request for tablet 6459af4a4180422a8eccf12fc862499e from peer 90365b2687854499bddaa63c08c0964f ({username='slave'} at 127.0.0.1:58756)
I20251026 09:10:53.619495 6543 tablet_copy_service.cc:161] P 3ed603c643ac44ca9908b171687d7d5c: Beginning new tablet copy session on tablet 6459af4a4180422a8eccf12fc862499e from peer 90365b2687854499bddaa63c08c0964f at {username='slave'} at 127.0.0.1:58756: session id = 90365b2687854499bddaa63c08c0964f-6459af4a4180422a8eccf12fc862499e
I20251026 09:10:53.620043 6543 tablet_copy_source_session.cc:215] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:53.620417 7536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6459af4a4180422a8eccf12fc862499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:53.621860 7536 tablet_copy_client.cc:806] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:53.621987 7536 tablet_copy_client.cc:670] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:53.622654 7537 tablet_copy_client.cc:1025] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:53.684100 7538 ts_tablet_manager.cc:933] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: Initiating tablet copy from peer cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:53.684360 7538 tablet_copy_client.cc:323] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42631
I20251026 09:10:53.684544 6618 tablet_copy_service.cc:140] P cfcab08da3db40e1974902b787ef6361: Received BeginTabletCopySession request for tablet ff5ceda9af754029a9b5ac636a17ea70 from peer 485b30c027fe4845b029de91594146fd ({username='slave'} at 127.0.0.1:44520)
I20251026 09:10:53.684600 6618 tablet_copy_service.cc:161] P cfcab08da3db40e1974902b787ef6361: Beginning new tablet copy session on tablet ff5ceda9af754029a9b5ac636a17ea70 from peer 485b30c027fe4845b029de91594146fd at {username='slave'} at 127.0.0.1:44520: session id = 485b30c027fe4845b029de91594146fd-ff5ceda9af754029a9b5ac636a17ea70
I20251026 09:10:53.685093 6618 tablet_copy_source_session.cc:215] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:53.685442 7538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff5ceda9af754029a9b5ac636a17ea70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:53.686650 7538 tablet_copy_client.cc:806] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:53.686825 7538 tablet_copy_client.cc:670] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:53.687430 7539 tablet_copy_client.cc:1025] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:53.697299 7540 ts_tablet_manager.cc:933] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: Initiating tablet copy from peer cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:53.697542 7540 tablet_copy_client.cc:323] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42631
I20251026 09:10:53.697731 6618 tablet_copy_service.cc:140] P cfcab08da3db40e1974902b787ef6361: Received BeginTabletCopySession request for tablet 22a8d92b27b14f89a4728a8f50db932b from peer 90365b2687854499bddaa63c08c0964f ({username='slave'} at 127.0.0.1:44510)
I20251026 09:10:53.697785 6618 tablet_copy_service.cc:161] P cfcab08da3db40e1974902b787ef6361: Beginning new tablet copy session on tablet 22a8d92b27b14f89a4728a8f50db932b from peer 90365b2687854499bddaa63c08c0964f at {username='slave'} at 127.0.0.1:44510: session id = 90365b2687854499bddaa63c08c0964f-22a8d92b27b14f89a4728a8f50db932b
I20251026 09:10:53.698230 6618 tablet_copy_source_session.cc:215] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:53.698611 7540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22a8d92b27b14f89a4728a8f50db932b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:53.700222 7540 tablet_copy_client.cc:806] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:53.700394 7540 tablet_copy_client.cc:670] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:53.701053 7541 tablet_copy_client.cc:1025] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:53.769351 6605 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:53.769567 7381 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:53.770680 6533 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:10:53.771188 7543 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:10:54.860136 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:54.860623 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:54.860747 6684 raft_consensus.cc:606] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:54.861038 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:54.861111 6684 raft_consensus.cc:606] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:10:54.861263 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 09:10:54.861327 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:55.085091 7381 raft_consensus.cc:993] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:55.085197 7381 raft_consensus.cc:1081] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:55.085364 6605 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47312
I20251026 09:10:55.085474 6605 raft_consensus.cc:493] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:55.085521 6605 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:55.086484 6605 raft_consensus.cc:515] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:55.086738 6605 leader_election.cc:290] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Requested vote from peers 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689), 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:10:55.086799 7088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: true dest_uuid: "485b30c027fe4845b029de91594146fd"
I20251026 09:10:55.086840 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:55.086998 7088 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:55.087024 6684 raft_consensus.cc:3055] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 09:10:55.087061 6684 raft_consensus.cc:740] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:55.087111 6684 consensus_queue.cc:260] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:55.087236 6684 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:55.087710 7088 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 4.
I20251026 09:10:55.087850 6684 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 4.
I20251026 09:10:55.087926 6575 leader_election.cc:304] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485b30c027fe4845b029de91594146fd, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:55.088016 7550 raft_consensus.cc:2804] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 09:10:55.088068 7550 raft_consensus.cc:697] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:55.088124 7550 consensus_queue.cc:237] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:55.088744 6427 catalog_manager.cc:5649] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 3 to 4, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:55.187259 7381 raft_consensus.cc:993] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:55.187398 7381 raft_consensus.cc:1081] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:55.187614 6532 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44252
I20251026 09:10:55.187705 6532 raft_consensus.cc:493] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:55.187741 6532 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:55.188570 6532 raft_consensus.cc:515] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:55.188730 6532 leader_election.cc:290] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 4 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:55.188912 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:55.188989 6605 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:55.188918 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:55.189035 6684 raft_consensus.cc:3055] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 09:10:55.189065 6684 raft_consensus.cc:740] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:55.189119 6684 consensus_queue.cc:260] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:55.189185 6684 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:55.189678 6605 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 4.
I20251026 09:10:55.189706 6684 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 4.
I20251026 09:10:55.189872 6499 leader_election.cc:304] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:55.189976 7549 raft_consensus.cc:2804] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Leader election won for term 4
I20251026 09:10:55.190135 7549 raft_consensus.cc:697] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:55.190193 7549 consensus_queue.cc:237] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:55.190852 6427 catalog_manager.cc:5649] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 3 to 4, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 4 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:55.622884 7536 tablet_copy_client.cc:538] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:55.623916 7536 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: Bootstrap starting.
I20251026 09:10:55.628657 7536 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=145 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:55.628886 7536 tablet_bootstrap.cc:492] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: Bootstrap complete.
I20251026 09:10:55.629005 7536 ts_tablet_manager.cc:1403] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 09:10:55.629189 7536 raft_consensus.cc:359] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:55.629261 7536 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 90365b2687854499bddaa63c08c0964f, State: Initialized, Role: LEARNER
I20251026 09:10:55.629284 7536 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 6.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:55.629396 7536 ts_tablet_manager.cc:1434] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:10:55.629595 6543 tablet_copy_service.cc:342] P 3ed603c643ac44ca9908b171687d7d5c: Request end of tablet copy session 90365b2687854499bddaa63c08c0964f-6459af4a4180422a8eccf12fc862499e received from {username='slave'} at 127.0.0.1:58756
I20251026 09:10:55.629670 6543 tablet_copy_service.cc:434] P 3ed603c643ac44ca9908b171687d7d5c: ending tablet copy session 90365b2687854499bddaa63c08c0964f-6459af4a4180422a8eccf12fc862499e on tablet 6459af4a4180422a8eccf12fc862499e with peer 90365b2687854499bddaa63c08c0964f
I20251026 09:10:55.631193 6684 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20251026 09:10:55.631422 7550 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:55.632676 7088 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20251026 09:10:55.633011 7550 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:55.687722 7538 tablet_copy_client.cc:538] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:55.688835 7538 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: Bootstrap starting.
I20251026 09:10:55.694506 7538 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=142 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:55.694790 7538 tablet_bootstrap.cc:492] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: Bootstrap complete.
I20251026 09:10:55.694934 7538 ts_tablet_manager.cc:1403] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20251026 09:10:55.695124 7538 raft_consensus.cc:359] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:55.695236 7538 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 485b30c027fe4845b029de91594146fd, State: Initialized, Role: LEARNER
I20251026 09:10:55.695273 7538 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:55.695438 7538 ts_tablet_manager.cc:1434] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.004s
I20251026 09:10:55.695695 6618 tablet_copy_service.cc:342] P cfcab08da3db40e1974902b787ef6361: Request end of tablet copy session 485b30c027fe4845b029de91594146fd-ff5ceda9af754029a9b5ac636a17ea70 received from {username='slave'} at 127.0.0.1:44520
I20251026 09:10:55.695756 6618 tablet_copy_service.cc:434] P cfcab08da3db40e1974902b787ef6361: ending tablet copy session 485b30c027fe4845b029de91594146fd-ff5ceda9af754029a9b5ac636a17ea70 on tablet ff5ceda9af754029a9b5ac636a17ea70 with peer 485b30c027fe4845b029de91594146fd
I20251026 09:10:55.700248 7088 raft_consensus.cc:1217] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 4 LEARNER]: Deduplicated request from leader. Original: 4.41->[4.42-4.42] Dedup: 4.42->[]
I20251026 09:10:55.701211 7540 tablet_copy_client.cc:538] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:55.702476 7540 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: Bootstrap starting.
I20251026 09:10:55.708847 7540 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=41 overwritten=0 applied=41 ignored=0} inserts{seen=149 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:55.709151 7540 tablet_bootstrap.cc:492] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: Bootstrap complete.
I20251026 09:10:55.709301 7540 ts_tablet_manager.cc:1403] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20251026 09:10:55.709508 7540 raft_consensus.cc:359] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:55.709641 7540 raft_consensus.cc:740] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 90365b2687854499bddaa63c08c0964f, State: Initialized, Role: LEARNER
I20251026 09:10:55.709695 7540 consensus_queue.cc:260] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 2.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: NON_VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: true } }
I20251026 09:10:55.709803 7540 ts_tablet_manager.cc:1434] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s
I20251026 09:10:55.710031 6618 tablet_copy_service.cc:342] P cfcab08da3db40e1974902b787ef6361: Request end of tablet copy session 90365b2687854499bddaa63c08c0964f-22a8d92b27b14f89a4728a8f50db932b received from {username='slave'} at 127.0.0.1:44510
I20251026 09:10:55.710088 6618 tablet_copy_service.cc:434] P cfcab08da3db40e1974902b787ef6361: ending tablet copy session 90365b2687854499bddaa63c08c0964f-22a8d92b27b14f89a4728a8f50db932b on tablet 22a8d92b27b14f89a4728a8f50db932b with peer 90365b2687854499bddaa63c08c0964f
I20251026 09:10:55.728955 6684 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 4 index: 39. (index mismatch)
I20251026 09:10:55.729207 7549 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:55.730358 6605 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 4 index: 39. (index mismatch)
I20251026 09:10:55.730860 7549 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:10:55.857669 7014 raft_consensus.cc:1217] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 6 LEARNER]: Deduplicated request from leader. Original: 6.44->[6.45-6.45] Dedup: 6.45->[]
I20251026 09:10:56.111876 7014 raft_consensus.cc:1217] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.40->[2.41-2.41] Dedup: 2.41->[]
I20251026 09:10:56.197602 7574 raft_consensus.cc:1064] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: attempting to promote NON_VOTER 485b30c027fe4845b029de91594146fd to VOTER
I20251026 09:10:56.197961 7574 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 40, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:56.198562 7088 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 4 LEARNER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 4 index: 43. (index mismatch)
I20251026 09:10:56.198622 6533 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 4 index: 43. (index mismatch)
I20251026 09:10:56.198626 6938 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 4 index: 43. (index mismatch)
I20251026 09:10:56.198774 7576 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:56.198936 7573 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:56.199059 7564 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:56.199971 7576 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Committing config change with OpId 4.43: config changed from index 42 to 43, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:56.200112 6533 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Committing config change with OpId 4.43: config changed from index 42 to 43, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:56.200488 6938 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Committing config change with OpId 4.43: config changed from index 42 to 43, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:56.201335 7088 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Committing config change with OpId 4.43: config changed from index 42 to 43, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:56.201586 6428 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 42 to 43, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:56.203269 6605 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 4.43, Last appended by leader: 40, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:56.203754 7088 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 43. Preceding OpId from leader: term: 4 index: 44. (index mismatch)
I20251026 09:10:56.203754 6938 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 4 index: 43. Preceding OpId from leader: term: 4 index: 44. (index mismatch)
I20251026 09:10:56.203967 7573 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:56.204089 7576 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:56.204840 7574 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Committing config change with OpId 4.44: config changed from index 43 to 44, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:56.205011 7088 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Committing config change with OpId 4.44: config changed from index 43 to 44, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:56.205861 6938 raft_consensus.cc:2955] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Committing config change with OpId 4.44: config changed from index 43 to 44, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:56.206894 6427 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd reported cstate change: config changed from index 43 to 44, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:10:56.207183 6417 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet ff5ceda9af754029a9b5ac636a17ea70 with cas_config_opid_index 43: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:56.208185 6518 tablet_service.cc:1552] Processing DeleteTablet for tablet ff5ceda9af754029a9b5ac636a17ea70 with delete_type TABLET_DATA_TOMBSTONED (TS 3ed603c643ac44ca9908b171687d7d5c not found in new config with opid_index 44) from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:56.208487 7584 tablet_replica.cc:333] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:10:56.208570 7584 raft_consensus.cc:2243] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:56.208688 7584 raft_consensus.cc:2272] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:56.209038 7584 ts_tablet_manager.cc:1916] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:56.211091 7584 ts_tablet_manager.cc:1929] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.43
I20251026 09:10:56.211167 7584 log.cc:1199] T ff5ceda9af754029a9b5ac636a17ea70 P 3ed603c643ac44ca9908b171687d7d5c: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/ff5ceda9af754029a9b5ac636a17ea70
I20251026 09:10:56.211616 6414 catalog_manager.cc:4985] TS 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705): tablet ff5ceda9af754029a9b5ac636a17ea70 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:56.354108 7549 raft_consensus.cc:1064] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: attempting to promote NON_VOTER 90365b2687854499bddaa63c08c0964f to VOTER
I20251026 09:10:56.354362 7549 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 6.45, Last appended by leader: 41, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:56.354921 6684 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 45. Preceding OpId from leader: term: 6 index: 46. (index mismatch)
I20251026 09:10:56.354996 6938 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 45. Preceding OpId from leader: term: 6 index: 46. (index mismatch)
I20251026 09:10:56.354921 7014 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 6 LEARNER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 45. Preceding OpId from leader: term: 6 index: 46. (index mismatch)
I20251026 09:10:56.355170 7549 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:10:56.355350 7588 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:10:56.356089 7549 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:10:56.356287 7585 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Committing config change with OpId 6.46: config changed from index 45 to 46, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.356439 6938 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Committing config change with OpId 6.46: config changed from index 45 to 46, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.356884 7014 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 6 FOLLOWER]: Committing config change with OpId 6.46: config changed from index 45 to 46, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.357919 6430 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f reported cstate change: config changed from index 45 to 46, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.358256 6684 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Committing config change with OpId 6.46: config changed from index 45 to 46, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.360311 6533 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 6.46, Last appended by leader: 41, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:56.360996 7014 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 46. Preceding OpId from leader: term: 6 index: 47. (index mismatch)
I20251026 09:10:56.361020 6938 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 6 index: 46. Preceding OpId from leader: term: 6 index: 47. (index mismatch)
I20251026 09:10:56.361258 7549 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251026 09:10:56.361368 7588 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251026 09:10:56.361991 7585 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Committing config change with OpId 6.47: config changed from index 46 to 47, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.362103 7014 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 6 FOLLOWER]: Committing config change with OpId 6.47: config changed from index 46 to 47, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.363054 6414 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 6459af4a4180422a8eccf12fc862499e with cas_config_opid_index 46: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:56.363039 6938 raft_consensus.cc:2955] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Committing config change with OpId 6.47: config changed from index 46 to 47, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.363435 6427 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 46 to 47, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New cstate: current_term: 6 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:56.364282 6665 tablet_service.cc:1552] Processing DeleteTablet for tablet 6459af4a4180422a8eccf12fc862499e with delete_type TABLET_DATA_TOMBSTONED (TS 6d20fbac1f1243a0b383c9383a1d0a70 not found in new config with opid_index 47) from {username='slave'} at 127.0.0.1:33620
I20251026 09:10:56.364642 7592 tablet_replica.cc:333] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:10:56.364709 7592 raft_consensus.cc:2243] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:56.364840 7592 raft_consensus.cc:2272] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:56.365098 7592 ts_tablet_manager.cc:1916] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:56.367130 7592 ts_tablet_manager.cc:1929] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.46
I20251026 09:10:56.367216 7592 log.cc:1199] T 6459af4a4180422a8eccf12fc862499e P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/6459af4a4180422a8eccf12fc862499e
I20251026 09:10:56.367638 6416 catalog_manager.cc:4985] TS 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689): tablet 6459af4a4180422a8eccf12fc862499e (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:56.526119 7576 raft_consensus.cc:1064] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: attempting to promote NON_VOTER 90365b2687854499bddaa63c08c0964f to VOTER
I20251026 09:10:56.526371 7576 consensus_queue.cc:237] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 2.41, Last appended by leader: 36, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:56.526952 6938 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 2 index: 42. (index mismatch)
I20251026 09:10:56.526993 6533 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 2 index: 42. (index mismatch)
I20251026 09:10:56.527132 7014 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [term 2 LEARNER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 2 index: 42. (index mismatch)
I20251026 09:10:56.527204 7564 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:56.527385 7574 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:56.527513 7564 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:56.528331 7573 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.42: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.528453 6938 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Committing config change with OpId 2.42: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.529495 7014 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Committing config change with OpId 2.42: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.529757 6533 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.42: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.529793 6429 catalog_manager.cc:5649] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed reported cstate change: config changed from index 41 to 42, 90365b2687854499bddaa63c08c0964f (127.3.14.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { replace: true } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.530949 6604 consensus_queue.cc:237] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 2.42, Last appended by leader: 36, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:10:56.531723 6938 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 42. Preceding OpId from leader: term: 2 index: 43. (index mismatch)
I20251026 09:10:56.531723 7014 raft_consensus.cc:1275] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 2 index: 42. Preceding OpId from leader: term: 2 index: 43. (index mismatch)
I20251026 09:10:56.531919 7574 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:56.532003 7564 consensus_queue.cc:1048] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:56.532501 7573 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Committing config change with OpId 2.43: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.532606 6938 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Committing config change with OpId 2.43: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.533802 6417 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 22a8d92b27b14f89a4728a8f50db932b with cas_config_opid_index 42: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:10:56.533836 6427 catalog_manager.cc:5649] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 reported cstate change: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New cstate: current_term: 2 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:56.533782 7014 raft_consensus.cc:2955] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Committing config change with OpId 2.43: config changed from index 42 to 43, VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } }
I20251026 09:10:56.534672 6518 tablet_service.cc:1552] Processing DeleteTablet for tablet 22a8d92b27b14f89a4728a8f50db932b with delete_type TABLET_DATA_TOMBSTONED (TS 3ed603c643ac44ca9908b171687d7d5c not found in new config with opid_index 43) from {username='slave'} at 127.0.0.1:44246
I20251026 09:10:56.534778 7584 tablet_replica.cc:333] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:10:56.534848 7584 raft_consensus.cc:2243] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:10:56.534948 7584 raft_consensus.cc:2272] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:10:56.535295 7584 ts_tablet_manager.cc:1916] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:10:56.536599 7584 ts_tablet_manager.cc:1929] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.42
I20251026 09:10:56.536665 7584 log.cc:1199] T 22a8d92b27b14f89a4728a8f50db932b P 3ed603c643ac44ca9908b171687d7d5c: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/22a8d92b27b14f89a4728a8f50db932b
I20251026 09:10:56.536999 6414 catalog_manager.cc:4985] TS 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705): tablet 22a8d92b27b14f89a4728a8f50db932b (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:10:56.861495 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:56.862074 6533 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:10:56.862171 6533 raft_consensus.cc:606] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Received request to transfer leadership to TS 6d20fbac1f1243a0b383c9383a1d0a70
I20251026 09:10:56.862634 6605 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "485b30c027fe4845b029de91594146fd"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:10:56.862730 6605 raft_consensus.cc:606] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Received request to transfer leadership to TS 485b30c027fe4845b029de91594146fd
I20251026 09:10:56.862957 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 09:10:56.863029 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:57.079005 7573 raft_consensus.cc:993] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: : Instructing follower 485b30c027fe4845b029de91594146fd to start an election
I20251026 09:10:57.079092 7573 raft_consensus.cc:1081] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Signalling peer 485b30c027fe4845b029de91594146fd to start an election
I20251026 09:10:57.079356 7088 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb"
dest_uuid: "485b30c027fe4845b029de91594146fd"
from {username='slave'} at 127.0.0.1:40432
I20251026 09:10:57.079434 7088 raft_consensus.cc:493] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:57.079457 7088 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:57.080164 7088 raft_consensus.cc:515] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:57.080327 7088 leader_election.cc:290] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [CANDIDATE]: Term 5 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:57.080513 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "485b30c027fe4845b029de91594146fd" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:57.080520 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9723f81063e4f23bb2300f2c7bd50eb" candidate_uuid: "485b30c027fe4845b029de91594146fd" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:57.080595 6605 raft_consensus.cc:3055] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 09:10:57.080698 6684 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:57.080711 6605 raft_consensus.cc:740] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:57.080778 6605 consensus_queue.cc:260] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:57.080919 6605 raft_consensus.cc:3060] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:57.081429 6684 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485b30c027fe4845b029de91594146fd in term 5.
I20251026 09:10:57.081513 6605 raft_consensus.cc:2468] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485b30c027fe4845b029de91594146fd in term 5.
I20251026 09:10:57.081665 7054 leader_election.cc:304] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485b30c027fe4845b029de91594146fd, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:10:57.081766 7416 raft_consensus.cc:2804] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Leader election won for term 5
I20251026 09:10:57.081970 7416 raft_consensus.cc:697] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 5 LEADER]: Becoming Leader. State: Replica: 485b30c027fe4845b029de91594146fd, State: Running, Role: LEADER
I20251026 09:10:57.082054 7416 consensus_queue.cc:237] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:57.082747 6427 catalog_manager.cc:5649] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd reported cstate change: term changed from 4 to 5, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 485b30c027fe4845b029de91594146fd (127.3.14.6). New cstate: current_term: 5 leader_uuid: "485b30c027fe4845b029de91594146fd" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:10:57.299844 7588 raft_consensus.cc:993] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:57.299939 7588 raft_consensus.cc:1081] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:10:57.300163 6684 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33632
I20251026 09:10:57.300259 6684 raft_consensus.cc:493] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:57.300294 6684 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:57.300997 6684 raft_consensus.cc:515] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:57.301168 6684 leader_election.cc:290] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 5 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631)
I20251026 09:10:57.301347 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:57.301432 6533 raft_consensus.cc:3055] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Stepping down as leader of term 4
I20251026 09:10:57.301415 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:57.301465 6533 raft_consensus.cc:740] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:57.301486 6605 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:57.301524 6533 consensus_queue.cc:260] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:57.301627 6533 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:10:57.302310 6533 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 5.
I20251026 09:10:57.302310 6605 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 5.
I20251026 09:10:57.302577 6650 leader_election.cc:304] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20fbac1f1243a0b383c9383a1d0a70, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:57.302701 7381 raft_consensus.cc:2804] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 09:10:57.302827 7381 raft_consensus.cc:697] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:57.302874 7381 consensus_queue.cc:237] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:57.303452 6427 catalog_manager.cc:5649] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 4 to 5, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 5 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } }
I20251026 09:10:57.507177 6684 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 5 index: 43. (index mismatch)
I20251026 09:10:57.507560 7416 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:57.508769 6605 raft_consensus.cc:1275] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Refusing update from remote peer 485b30c027fe4845b029de91594146fd: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 5 index: 43. (index mismatch)
I20251026 09:10:57.509084 7416 consensus_queue.cc:1048] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:10:57.538018 7014 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 5.43, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:57.538642 6533 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Refusing update from remote peer 90365b2687854499bddaa63c08c0964f: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 5 index: 44. (index mismatch)
I20251026 09:10:57.538777 6605 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Refusing update from remote peer 90365b2687854499bddaa63c08c0964f: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 5 index: 44. (index mismatch)
W20251026 09:10:57.538777 6980 consensus_peers.cc:597] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f -> Peer 485b30c027fe4845b029de91594146fd (127.3.14.6:35539): Couldn't send request to peer 485b30c027fe4845b029de91594146fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21bd52ca927842ccab72fc70494128d0. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:57.538913 7615 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:57.539069 7614 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:57.539690 7615 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 5 LEADER]: Committing config change with OpId 5.44: config changed from index 42 to 44, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:57.539789 6533 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Committing config change with OpId 5.44: config changed from index 42 to 44, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:57.539938 6605 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Committing config change with OpId 5.44: config changed from index 42 to 44, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:57.540969 6427 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 42 to 44, NON_VOTER 485b30c027fe4845b029de91594146fd (127.3.14.6) added. New cstate: current_term: 5 leader_uuid: "90365b2687854499bddaa63c08c0964f" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } } }
I20251026 09:10:57.541297 6684 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 3.43, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:10:57.541922 7014 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 43. Preceding OpId from leader: term: 3 index: 44. (index mismatch)
I20251026 09:10:57.541988 7088 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 43. Preceding OpId from leader: term: 3 index: 44. (index mismatch)
I20251026 09:10:57.542147 7604 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:57.542240 7381 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
W20251026 09:10:57.542382 6650 consensus_peers.cc:597] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 -> Peer 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841): Couldn't send request to peer 26c251f040b9472a92191f35891e8eed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bb5819b36e244d4b0ab2c0e110e51e9. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:57.542963 7381 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.44: config changed from index 43 to 44, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:57.543196 7014 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Committing config change with OpId 3.44: config changed from index 43 to 44, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:57.544102 6429 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 43 to 44, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:57.544238 7088 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Committing config change with OpId 3.44: config changed from index 43 to 44, NON_VOTER 26c251f040b9472a92191f35891e8eed (127.3.14.4) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } } }
I20251026 09:10:57.544375 6938 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 2.43, Last appended by leader: 40, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: NON_VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: true } }
I20251026 09:10:57.544955 6684 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 43. Preceding OpId from leader: term: 2 index: 44. (index mismatch)
I20251026 09:10:57.545025 7014 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 43. Preceding OpId from leader: term: 2 index: 44. (index mismatch)
I20251026 09:10:57.545176 7489 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
W20251026 09:10:57.545327 6901 consensus_peers.cc:597] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed -> Peer 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705): Couldn't send request to peer 3ed603c643ac44ca9908b171687d7d5c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:10:57.545499 7623 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251026 09:10:57.546168 7489 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 LEADER]: Committing config change with OpId 2.44: config changed from index 43 to 44, NON_VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: NON_VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: true } } }
I20251026 09:10:57.546255 7014 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Committing config change with OpId 2.44: config changed from index 43 to 44, NON_VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: NON_VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: true } } }
I20251026 09:10:57.547170 6684 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.44: config changed from index 43 to 44, NON_VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: NON_VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: true } } }
I20251026 09:10:57.547185 6430 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f reported cstate change: config changed from index 43 to 44, NON_VOTER 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) added. New cstate: current_term: 2 leader_uuid: "26c251f040b9472a92191f35891e8eed" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: NON_VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: true } } }
I20251026 09:10:57.724552 6533 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 5 index: 40. (index mismatch)
I20251026 09:10:57.724820 7381 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:57.726082 6605 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 5 index: 40. (index mismatch)
I20251026 09:10:57.726336 7619 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20251026 09:10:57.986522 7630 ts_tablet_manager.cc:933] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: Initiating tablet copy from peer 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007)
I20251026 09:10:57.986863 7630 tablet_copy_client.cc:323] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.5:40007
I20251026 09:10:57.987074 7024 tablet_copy_service.cc:140] P 90365b2687854499bddaa63c08c0964f: Received BeginTabletCopySession request for tablet 21bd52ca927842ccab72fc70494128d0 from peer 485b30c027fe4845b029de91594146fd ({username='slave'} at 127.0.0.1:34766)
I20251026 09:10:57.987139 7024 tablet_copy_service.cc:161] P 90365b2687854499bddaa63c08c0964f: Beginning new tablet copy session on tablet 21bd52ca927842ccab72fc70494128d0 from peer 485b30c027fe4845b029de91594146fd at {username='slave'} at 127.0.0.1:34766: session id = 485b30c027fe4845b029de91594146fd-21bd52ca927842ccab72fc70494128d0
I20251026 09:10:57.987764 7024 tablet_copy_source_session.cc:215] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:57.988135 7630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21bd52ca927842ccab72fc70494128d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:57.989847 7630 tablet_copy_client.cc:806] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:57.989984 7630 tablet_copy_client.cc:670] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:57.990586 7631 tablet_copy_client.cc:1025] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:58.104612 7633 ts_tablet_manager.cc:933] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: Initiating tablet copy from peer 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:58.104840 7633 tablet_copy_client.cc:323] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:39689
I20251026 09:10:58.105031 6694 tablet_copy_service.cc:140] P 6d20fbac1f1243a0b383c9383a1d0a70: Received BeginTabletCopySession request for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 from peer 26c251f040b9472a92191f35891e8eed ({username='slave'} at 127.0.0.1:44952)
I20251026 09:10:58.105093 6694 tablet_copy_service.cc:161] P 6d20fbac1f1243a0b383c9383a1d0a70: Beginning new tablet copy session on tablet 4bb5819b36e244d4b0ab2c0e110e51e9 from peer 26c251f040b9472a92191f35891e8eed at {username='slave'} at 127.0.0.1:44952: session id = 26c251f040b9472a92191f35891e8eed-4bb5819b36e244d4b0ab2c0e110e51e9
I20251026 09:10:58.105542 6694 tablet_copy_source_session.cc:215] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:58.105914 7633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb5819b36e244d4b0ab2c0e110e51e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:58.108678 7633 tablet_copy_client.cc:806] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:58.108822 7633 tablet_copy_client.cc:670] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:58.109503 7634 tablet_copy_client.cc:1025] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:58.127995 7636 ts_tablet_manager.cc:933] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Initiating tablet copy from peer 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841)
I20251026 09:10:58.128243 7636 tablet_copy_client.cc:323] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.4:39841
I20251026 09:10:58.128567 6948 tablet_copy_service.cc:140] P 26c251f040b9472a92191f35891e8eed: Received BeginTabletCopySession request for tablet ad759f01ba6c4154968854b4b9378651 from peer 3ed603c643ac44ca9908b171687d7d5c ({username='slave'} at 127.0.0.1:53940)
I20251026 09:10:58.128635 6948 tablet_copy_service.cc:161] P 26c251f040b9472a92191f35891e8eed: Beginning new tablet copy session on tablet ad759f01ba6c4154968854b4b9378651 from peer 3ed603c643ac44ca9908b171687d7d5c at {username='slave'} at 127.0.0.1:53940: session id = 3ed603c643ac44ca9908b171687d7d5c-ad759f01ba6c4154968854b4b9378651
I20251026 09:10:58.129375 6948 tablet_copy_source_session.cc:215] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:10:58.129648 7636 ts_tablet_manager.cc:1916] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20251026 09:10:58.131381 7636 ts_tablet_manager.cc:1929] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.42
I20251026 09:10:58.131508 7636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad759f01ba6c4154968854b4b9378651. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:10:58.132297 7636 tablet_copy_client.cc:806] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: tablet copy: Starting download of 0 data blocks...
I20251026 09:10:58.132464 7636 tablet_copy_client.cc:670] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: tablet copy: Starting download of 1 WAL segments...
I20251026 09:10:58.133082 7637 tablet_copy_client.cc:1025] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: tablet copy: Injecting latency into file download: 2000
I20251026 09:10:58.863195 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:10:58.863721 6684 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:10:58.863817 6684 raft_consensus.cc:606] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:10:58.872186 7088 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90"
dest_uuid: "485b30c027fe4845b029de91594146fd"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:43450
I20251026 09:10:58.872326 7088 raft_consensus.cc:606] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 3 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:10:58.872485 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 09:10:58.872566 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:10:58.958628 7642 raft_consensus.cc:993] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:58.958741 7642 raft_consensus.cc:1081] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:10:58.958920 6533 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90072194423a4007b85cbc3f31889400"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44252
I20251026 09:10:58.959038 6533 raft_consensus.cc:493] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:58.959071 6533 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Advancing to term 6
I20251026 09:10:58.960044 6533 raft_consensus.cc:515] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:58.960211 6533 leader_election.cc:290] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 6 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:10:58.960319 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 6 candidate_status { last_received { term: 5 index: 40 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:10:58.960407 6605 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Advancing to term 6
I20251026 09:10:58.960443 6684 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90072194423a4007b85cbc3f31889400" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 6 candidate_status { last_received { term: 5 index: 40 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:10:58.960505 6684 raft_consensus.cc:3055] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 09:10:58.960533 6684 raft_consensus.cc:740] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:10:58.960590 6684 consensus_queue.cc:260] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 5.40, Last appended by leader: 40, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:58.960666 6684 raft_consensus.cc:3060] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Advancing to term 6
I20251026 09:10:58.961225 6605 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 6.
I20251026 09:10:58.961398 6499 leader_election.cc:304] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:58.961448 6684 raft_consensus.cc:2468] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 6.
I20251026 09:10:58.961506 7588 raft_consensus.cc:2804] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 6 FOLLOWER]: Leader election won for term 6
I20251026 09:10:58.961627 7588 raft_consensus.cc:697] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:10:58.961694 7588 consensus_queue.cc:237] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 5.40, Last appended by leader: 40, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:10:58.962180 6429 catalog_manager.cc:5649] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 5 to 6, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 6 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:59.294232 7632 raft_consensus.cc:993] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:59.294349 7632 raft_consensus.cc:1081] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 3 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:10:59.294528 6605 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:44520
I20251026 09:10:59.294642 6605 raft_consensus.cc:493] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:10:59.294668 6605 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:59.295398 6605 raft_consensus.cc:515] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:59.295578 6605 leader_election.cc:290] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:10:59.295703 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:10:59.295805 6533 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:59.296669 6533 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 4.
I20251026 09:10:59.296782 7088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: true dest_uuid: "485b30c027fe4845b029de91594146fd"
I20251026 09:10:59.296821 6572 leader_election.cc:304] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:10:59.296844 7088 raft_consensus.cc:3055] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 3 LEADER]: Stepping down as leader of term 3
I20251026 09:10:59.296873 7088 raft_consensus.cc:740] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 485b30c027fe4845b029de91594146fd, State: Running, Role: LEADER
I20251026 09:10:59.296897 7573 raft_consensus.cc:2804] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 09:10:59.296933 7088 consensus_queue.cc:260] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:59.297011 7088 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:10:59.297067 7573 raft_consensus.cc:697] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:10:59.297114 7573 consensus_queue.cc:237] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:10:59.297641 6429 catalog_manager.cc:5649] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 3 to 4, leader changed from 485b30c027fe4845b029de91594146fd (127.3.14.6) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:10:59.297871 7088 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 4.
I20251026 09:10:59.396956 6605 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 5 index: 40. Preceding OpId from leader: term: 6 index: 41. (index mismatch)
I20251026 09:10:59.397222 7588 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:59.398586 6684 raft_consensus.cc:1275] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 5 index: 40. Preceding OpId from leader: term: 6 index: 41. (index mismatch)
I20251026 09:10:59.398842 7645 consensus_queue.cc:1048] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251026 09:10:59.871657 6533 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20251026 09:10:59.871897 7573 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:59.873276 7088 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20251026 09:10:59.873605 7662 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251026 09:10:59.990870 7630 tablet_copy_client.cc:538] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:10:59.992197 7630 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: Bootstrap starting.
I20251026 09:10:59.997701 7630 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=143 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:10:59.997964 7630 tablet_bootstrap.cc:492] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: Bootstrap complete.
I20251026 09:10:59.998109 7630 ts_tablet_manager.cc:1403] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251026 09:10:59.998320 7630 raft_consensus.cc:359] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:59.998427 7630 raft_consensus.cc:740] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 485b30c027fe4845b029de91594146fd, State: Initialized, Role: LEARNER
I20251026 09:10:59.998476 7630 consensus_queue.cc:260] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 5.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: NON_VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: true } }
I20251026 09:10:59.998649 7630 ts_tablet_manager.cc:1434] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s
I20251026 09:10:59.998862 7024 tablet_copy_service.cc:342] P 90365b2687854499bddaa63c08c0964f: Request end of tablet copy session 485b30c027fe4845b029de91594146fd-21bd52ca927842ccab72fc70494128d0 received from {username='slave'} at 127.0.0.1:34766
I20251026 09:10:59.998916 7024 tablet_copy_service.cc:434] P 90365b2687854499bddaa63c08c0964f: ending tablet copy session 485b30c027fe4845b029de91594146fd-21bd52ca927842ccab72fc70494128d0 on tablet 21bd52ca927842ccab72fc70494128d0 with peer 485b30c027fe4845b029de91594146fd
I20251026 09:11:00.005412 7088 raft_consensus.cc:1217] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [term 5 LEARNER]: Deduplicated request from leader. Original: 5.43->[5.44-5.44] Dedup: 5.44->[]
I20251026 09:11:00.109871 7633 tablet_copy_client.cc:538] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:00.111054 7633 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: Bootstrap starting.
I20251026 09:11:00.117522 7633 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=140 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:00.117839 7633 tablet_bootstrap.cc:492] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: Bootstrap complete.
I20251026 09:11:00.118171 7633 ts_tablet_manager.cc:1403] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: Time spent bootstrapping tablet: real 0.007s user 0.008s sys 0.000s
I20251026 09:11:00.118579 7633 raft_consensus.cc:359] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:11:00.118685 7633 raft_consensus.cc:740] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Initialized, Role: LEARNER
I20251026 09:11:00.118724 7633 consensus_queue.cc:260] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 3.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: true } }
I20251026 09:11:00.118834 7633 ts_tablet_manager.cc:1434] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:11:00.119058 6694 tablet_copy_service.cc:342] P 6d20fbac1f1243a0b383c9383a1d0a70: Request end of tablet copy session 26c251f040b9472a92191f35891e8eed-4bb5819b36e244d4b0ab2c0e110e51e9 received from {username='slave'} at 127.0.0.1:44952
I20251026 09:11:00.119131 6694 tablet_copy_service.cc:434] P 6d20fbac1f1243a0b383c9383a1d0a70: ending tablet copy session 26c251f040b9472a92191f35891e8eed-4bb5819b36e244d4b0ab2c0e110e51e9 on tablet 4bb5819b36e244d4b0ab2c0e110e51e9 with peer 26c251f040b9472a92191f35891e8eed
I20251026 09:11:00.133303 7636 tablet_copy_client.cc:538] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:00.134531 7636 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap starting.
I20251026 09:11:00.141144 7636 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=44 ignored=0} inserts{seen=148 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:00.141456 7636 tablet_bootstrap.cc:492] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Bootstrap complete.
I20251026 09:11:00.141640 7636 ts_tablet_manager.cc:1403] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.001s
I20251026 09:11:00.141865 7636 raft_consensus.cc:359] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: NON_VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: true } }
I20251026 09:11:00.141969 7636 raft_consensus.cc:740] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Initialized, Role: LEARNER
I20251026 09:11:00.142004 7636 consensus_queue.cc:260] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 2.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: NON_VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: true } }
I20251026 09:11:00.142290 7636 ts_tablet_manager.cc:1434] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:11:00.142478 6948 tablet_copy_service.cc:342] P 26c251f040b9472a92191f35891e8eed: Request end of tablet copy session 3ed603c643ac44ca9908b171687d7d5c-ad759f01ba6c4154968854b4b9378651 received from {username='slave'} at 127.0.0.1:53940
I20251026 09:11:00.142531 6948 tablet_copy_service.cc:434] P 26c251f040b9472a92191f35891e8eed: ending tablet copy session 3ed603c643ac44ca9908b171687d7d5c-ad759f01ba6c4154968854b4b9378651 on tablet ad759f01ba6c4154968854b4b9378651 with peer 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:11:00.238202 6533 raft_consensus.cc:1217] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.43->[2.44-2.44] Dedup: 2.44->[]
I20251026 09:11:00.275388 6937 raft_consensus.cc:1217] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [term 3 LEARNER]: Deduplicated request from leader. Original: 3.43->[3.44-3.44] Dedup: 3.44->[]
I20251026 09:11:00.571700 7615 raft_consensus.cc:1064] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: attempting to promote NON_VOTER 485b30c027fe4845b029de91594146fd to VOTER
I20251026 09:11:00.571945 7615 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 5.44, Last appended by leader: 42, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:00.572556 6605 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Refusing update from remote peer 90365b2687854499bddaa63c08c0964f: Log matching property violated. Preceding OpId in replica: term: 5 index: 44. Preceding OpId from leader: term: 5 index: 45. (index mismatch)
I20251026 09:11:00.572563 7088 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [term 5 LEARNER]: Refusing update from remote peer 90365b2687854499bddaa63c08c0964f: Log matching property violated. Preceding OpId in replica: term: 5 index: 44. Preceding OpId from leader: term: 5 index: 45. (index mismatch)
I20251026 09:11:00.572636 6533 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Refusing update from remote peer 90365b2687854499bddaa63c08c0964f: Log matching property violated. Preceding OpId in replica: term: 5 index: 44. Preceding OpId from leader: term: 5 index: 45. (index mismatch)
I20251026 09:11:00.572798 7673 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:00.572927 7670 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:00.573051 7673 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:00.574066 7672 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 5 LEADER]: Committing config change with OpId 5.45: config changed from index 44 to 45, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:11:00.574215 6605 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Committing config change with OpId 5.45: config changed from index 44 to 45, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:11:00.574272 7088 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Committing config change with OpId 5.45: config changed from index 44 to 45, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:11:00.575484 6429 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd reported cstate change: config changed from index 44 to 45, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "90365b2687854499bddaa63c08c0964f" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:11:00.576066 6533 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Committing config change with OpId 5.45: config changed from index 44 to 45, 485b30c027fe4845b029de91594146fd (127.3.14.6) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } attrs { replace: true } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:11:00.577134 7014 consensus_queue.cc:237] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 5.45, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:00.577680 7088 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Refusing update from remote peer 90365b2687854499bddaa63c08c0964f: Log matching property violated. Preceding OpId in replica: term: 5 index: 45. Preceding OpId from leader: term: 5 index: 46. (index mismatch)
I20251026 09:11:00.577754 6533 raft_consensus.cc:1275] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Refusing update from remote peer 90365b2687854499bddaa63c08c0964f: Log matching property violated. Preceding OpId in replica: term: 5 index: 45. Preceding OpId from leader: term: 5 index: 46. (index mismatch)
I20251026 09:11:00.578054 7615 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:11:00.578169 7677 consensus_queue.cc:1048] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:11:00.578882 7677 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 5 LEADER]: Committing config change with OpId 5.46: config changed from index 45 to 46, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:11:00.579161 7088 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Committing config change with OpId 5.46: config changed from index 45 to 46, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:11:00.580101 6473 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 21bd52ca927842ccab72fc70494128d0 with cas_config_opid_index 45: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:11:00.580289 6427 catalog_manager.cc:5649] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd reported cstate change: config changed from index 45 to 46, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New cstate: current_term: 5 leader_uuid: "90365b2687854499bddaa63c08c0964f" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:11:00.580646 6533 raft_consensus.cc:2955] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Committing config change with OpId 5.46: config changed from index 45 to 46, VOTER cfcab08da3db40e1974902b787ef6361 (127.3.14.2) evicted. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } }
I20251026 09:11:00.582149 6596 tablet_service.cc:1552] Processing DeleteTablet for tablet 21bd52ca927842ccab72fc70494128d0 with delete_type TABLET_DATA_TOMBSTONED (TS cfcab08da3db40e1974902b787ef6361 not found in new config with opid_index 46) from {username='slave'} at 127.0.0.1:47308
I20251026 09:11:00.582449 7682 tablet_replica.cc:333] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:11:00.582567 7682 raft_consensus.cc:2243] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:00.582811 7682 raft_consensus.cc:2272] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:00.583073 7682 ts_tablet_manager.cc:1916] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:11:00.585201 7682 ts_tablet_manager.cc:1929] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.45
I20251026 09:11:00.585276 7682 log.cc:1199] T 21bd52ca927842ccab72fc70494128d0 P cfcab08da3db40e1974902b787ef6361: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/21bd52ca927842ccab72fc70494128d0
I20251026 09:11:00.585675 6417 catalog_manager.cc:4985] TS cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631): tablet 21bd52ca927842ccab72fc70494128d0 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:11:00.673728 7619 raft_consensus.cc:1064] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: attempting to promote NON_VOTER 26c251f040b9472a92191f35891e8eed to VOTER
I20251026 09:11:00.673976 7619 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 3.44, Last appended by leader: 37, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:11:00.674641 7088 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 44. Preceding OpId from leader: term: 3 index: 45. (index mismatch)
I20251026 09:11:00.674659 6937 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [term 3 LEARNER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 44. Preceding OpId from leader: term: 3 index: 45. (index mismatch)
I20251026 09:11:00.674713 7014 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 44. Preceding OpId from leader: term: 3 index: 45. (index mismatch)
I20251026 09:11:00.674897 7686 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:00.675069 7685 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:00.675484 7683 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:00.676312 7683 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.45: config changed from index 44 to 45, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:11:00.676528 6937 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Committing config change with OpId 3.45: config changed from index 44 to 45, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:11:00.676537 7014 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Committing config change with OpId 3.45: config changed from index 44 to 45, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:11:00.677665 7088 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Committing config change with OpId 3.45: config changed from index 44 to 45, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:11:00.677794 6427 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: config changed from index 44 to 45, 26c251f040b9472a92191f35891e8eed (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:11:00.678762 6681 consensus_queue.cc:237] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 3.45, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } }
I20251026 09:11:00.679220 7088 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 45. Preceding OpId from leader: term: 3 index: 46. (index mismatch)
I20251026 09:11:00.679287 6937 raft_consensus.cc:1275] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 3 index: 45. Preceding OpId from leader: term: 3 index: 46. (index mismatch)
I20251026 09:11:00.679524 7619 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:11:00.679641 7686 consensus_queue.cc:1048] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:11:00.680121 7619 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Committing config change with OpId 3.46: config changed from index 45 to 46, VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) evicted. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:11:00.680246 7088 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Committing config change with OpId 3.46: config changed from index 45 to 46, VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) evicted. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:11:00.681030 6937 raft_consensus.cc:2955] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Committing config change with OpId 3.46: config changed from index 45 to 46, VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) evicted. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:11:00.681213 6429 catalog_manager.cc:5649] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd reported cstate change: config changed from index 45 to 46, VOTER 90365b2687854499bddaa63c08c0964f (127.3.14.5) evicted. New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } }
I20251026 09:11:00.682081 6416 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 with cas_config_opid_index 45: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:11:00.685137 7004 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bb5819b36e244d4b0ab2c0e110e51e9 with delete_type TABLET_DATA_TOMBSTONED (TS 90365b2687854499bddaa63c08c0964f not found in new config with opid_index 46) from {username='slave'} at 127.0.0.1:52332
I20251026 09:11:00.685444 7692 tablet_replica.cc:333] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: stopping tablet replica
I20251026 09:11:00.685518 7692 raft_consensus.cc:2243] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:00.685664 7692 raft_consensus.cc:2272] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:00.685904 7692 ts_tablet_manager.cc:1916] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:11:00.687427 7692 ts_tablet_manager.cc:1929] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.45
I20251026 09:11:00.687516 7692 log.cc:1199] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 90365b2687854499bddaa63c08c0964f: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-4-root/wals/4bb5819b36e244d4b0ab2c0e110e51e9
I20251026 09:11:00.687916 6414 catalog_manager.cc:4985] TS 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007): tablet 4bb5819b36e244d4b0ab2c0e110e51e9 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:11:00.827586 7693 raft_consensus.cc:1064] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: attempting to promote NON_VOTER 3ed603c643ac44ca9908b171687d7d5c to VOTER
I20251026 09:11:00.827936 7693 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 2.44, Last appended by leader: 40, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } }
I20251026 09:11:00.828589 7014 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 44. Preceding OpId from leader: term: 2 index: 45. (index mismatch)
I20251026 09:11:00.828589 6533 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEARNER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 44. Preceding OpId from leader: term: 2 index: 45. (index mismatch)
I20251026 09:11:00.828758 6681 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 44. Preceding OpId from leader: term: 2 index: 45. (index mismatch)
I20251026 09:11:00.828884 7651 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:00.829031 7693 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:00.829213 7696 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:00.829964 7696 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 LEADER]: Committing config change with OpId 2.45: config changed from index 44 to 45, 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } } }
I20251026 09:11:00.830091 6533 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.45: config changed from index 44 to 45, 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } } }
I20251026 09:11:00.831007 6681 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Committing config change with OpId 2.45: config changed from index 44 to 45, 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } } }
I20251026 09:11:00.831231 6430 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 44 to 45, 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "26c251f040b9472a92191f35891e8eed" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } } }
I20251026 09:11:00.831805 7014 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Committing config change with OpId 2.45: config changed from index 44 to 45, 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) changed from NON_VOTER to VOTER. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } attrs { replace: true } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } } }
I20251026 09:11:00.833529 6938 consensus_queue.cc:237] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 2.45, Last appended by leader: 40, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } }
I20251026 09:11:00.833887 6533 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 45. Preceding OpId from leader: term: 2 index: 46. (index mismatch)
I20251026 09:11:00.833953 7014 raft_consensus.cc:1275] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 2 index: 45. Preceding OpId from leader: term: 2 index: 46. (index mismatch)
I20251026 09:11:00.834077 7651 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:11:00.834177 7695 consensus_queue.cc:1048] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:11:00.834880 7693 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 LEADER]: Committing config change with OpId 2.46: config changed from index 45 to 46, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } } }
I20251026 09:11:00.834998 6533 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Committing config change with OpId 2.46: config changed from index 45 to 46, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } } }
I20251026 09:11:00.834998 7014 raft_consensus.cc:2955] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Committing config change with OpId 2.46: config changed from index 45 to 46, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } } }
I20251026 09:11:00.835922 6477 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet ad759f01ba6c4154968854b4b9378651 with cas_config_opid_index 45: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:11:00.836061 6427 catalog_manager.cc:5649] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: config changed from index 45 to 46, VOTER 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) evicted. New cstate: current_term: 2 leader_uuid: "26c251f040b9472a92191f35891e8eed" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } attrs { promote: false } } }
I20251026 09:11:00.837040 6665 tablet_service.cc:1552] Processing DeleteTablet for tablet ad759f01ba6c4154968854b4b9378651 with delete_type TABLET_DATA_TOMBSTONED (TS 6d20fbac1f1243a0b383c9383a1d0a70 not found in new config with opid_index 46) from {username='slave'} at 127.0.0.1:33620
I20251026 09:11:00.837319 7700 tablet_replica.cc:333] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:11:00.837388 7700 raft_consensus.cc:2243] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:00.837466 7700 raft_consensus.cc:2272] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:00.837715 7700 ts_tablet_manager.cc:1916] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:11:00.838922 7700 ts_tablet_manager.cc:1929] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.45
I20251026 09:11:00.838976 7700 log.cc:1199] T ad759f01ba6c4154968854b4b9378651 P 6d20fbac1f1243a0b383c9383a1d0a70: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/ad759f01ba6c4154968854b4b9378651
I20251026 09:11:00.839356 6416 catalog_manager.cc:4985] TS 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689): tablet ad759f01ba6c4154968854b4b9378651 (table test-workload [id=801ba6b94b4647619e2c65accdea7862]) successfully deleted
I20251026 09:11:00.872757 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:00.873360 6605 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:11:00.873481 6605 raft_consensus.cc:606] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Received request to transfer leadership to TS 3ed603c643ac44ca9908b171687d7d5c
I20251026 09:11:00.873952 6533 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:11:00.874030 6533 raft_consensus.cc:606] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Received request to transfer leadership to TS 6d20fbac1f1243a0b383c9383a1d0a70
I20251026 09:11:00.874219 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 09:11:00.874294 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:00.911767 7669 raft_consensus.cc:993] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:11:00.911859 7669 raft_consensus.cc:1081] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Signalling peer 3ed603c643ac44ca9908b171687d7d5c to start an election
I20251026 09:11:00.912058 6533 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
from {username='slave'} at 127.0.0.1:44258
I20251026 09:11:00.912154 6533 raft_consensus.cc:493] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:11:00.912190 6533 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:11:00.912932 6533 raft_consensus.cc:515] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:00.913123 6533 leader_election.cc:290] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 5 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:11:00.913338 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:11:00.913347 7088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac1d836cf54d42c39eb2d8279fd17f90" candidate_uuid: "3ed603c643ac44ca9908b171687d7d5c" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: true dest_uuid: "485b30c027fe4845b029de91594146fd"
I20251026 09:11:00.913429 6605 raft_consensus.cc:3055] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 09:11:00.913448 7088 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:11:00.913450 6605 raft_consensus.cc:740] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:11:00.913543 6605 consensus_queue.cc:260] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:00.913635 6605 raft_consensus.cc:3060] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:11:00.914149 7088 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 5.
I20251026 09:11:00.914245 6605 raft_consensus.cc:2468] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed603c643ac44ca9908b171687d7d5c in term 5.
I20251026 09:11:00.914357 6499 leader_election.cc:304] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 485b30c027fe4845b029de91594146fd; no voters:
I20251026 09:11:00.914472 7665 raft_consensus.cc:2804] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Leader election won for term 5
I20251026 09:11:00.914592 7665 raft_consensus.cc:697] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 5 LEADER]: Becoming Leader. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:11:00.914652 7665 consensus_queue.cc:237] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:00.915200 6428 catalog_manager.cc:5649] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c reported cstate change: term changed from 4 to 5, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1). New cstate: current_term: 5 leader_uuid: "3ed603c643ac44ca9908b171687d7d5c" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:01.227241 7665 raft_consensus.cc:993] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:11:01.227406 7665 raft_consensus.cc:1081] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Signalling peer 6d20fbac1f1243a0b383c9383a1d0a70 to start an election
I20251026 09:11:01.227630 6681 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
from {username='slave'} at 127.0.0.1:33632
I20251026 09:11:01.227731 6681 raft_consensus.cc:493] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:11:01.227768 6681 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:11:01.228474 6681 raft_consensus.cc:515] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:11:01.228653 6681 leader_election.cc:290] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705)
I20251026 09:11:01.228895 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:11:01.228895 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:11:01.229009 6533 raft_consensus.cc:3055] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:11:01.228998 6605 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:11:01.229066 6533 raft_consensus.cc:740] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:11:01.229140 6533 consensus_queue.cc:260] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:11:01.229234 6533 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:11:01.229780 6605 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:11:01.229815 6533 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20fbac1f1243a0b383c9383a1d0a70 in term 3.
I20251026 09:11:01.229975 6647 leader_election.cc:304] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, 6d20fbac1f1243a0b383c9383a1d0a70; no voters:
I20251026 09:11:01.230068 7686 raft_consensus.cc:2804] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:11:01.230191 7686 raft_consensus.cc:697] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Leader. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:11:01.230253 7686 consensus_queue.cc:237] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 2.37, Last appended by leader: 37, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:11:01.230876 6428 catalog_manager.cc:5649] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 reported cstate change: term changed from 2 to 3, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3). New cstate: current_term: 3 leader_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: HEALTHY } } }
I20251026 09:11:01.445348 6605 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 5 index: 43. (index mismatch)
I20251026 09:11:01.445667 7665 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:11:01.446954 7088 raft_consensus.cc:1275] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Refusing update from remote peer 3ed603c643ac44ca9908b171687d7d5c: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 5 index: 43. (index mismatch)
I20251026 09:11:01.447196 7709 consensus_queue.cc:1048] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251026 09:11:01.665756 6533 raft_consensus.cc:1275] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:11:01.666029 7686 consensus_queue.cc:1048] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:11:01.667244 6605 raft_consensus.cc:1275] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Refusing update from remote peer 6d20fbac1f1243a0b383c9383a1d0a70: Log matching property violated. Preceding OpId in replica: term: 2 index: 37. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251026 09:11:01.667605 7714 consensus_queue.cc:1048] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251026 09:11:02.874480 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:02.875079 6533 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
mode: GRACEFUL
new_leader_uuid: "26c251f040b9472a92191f35891e8eed"
from {username='slave'} at 127.0.0.1:44286
I20251026 09:11:02.875232 6533 raft_consensus.cc:606] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Received request to transfer leadership to TS 26c251f040b9472a92191f35891e8eed
I20251026 09:11:02.875722 6681 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16"
dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:33660
I20251026 09:11:02.875818 6681 raft_consensus.cc:606] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:11:02.876041 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 09:11:02.876092 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:03.061810 7717 raft_consensus.cc:993] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:11:03.061898 7717 raft_consensus.cc:1081] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:11:03.062090 6605 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:47312
I20251026 09:11:03.062180 6605 raft_consensus.cc:493] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:11:03.062212 6605 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:11:03.062901 6605 raft_consensus.cc:515] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:11:03.063051 6605 leader_election.cc:290] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3:39689)
I20251026 09:11:03.063217 6681 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: true dest_uuid: "6d20fbac1f1243a0b383c9383a1d0a70"
I20251026 09:11:03.063244 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65f6e7109bc4eda8d83d836786c4d16" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:11:03.063293 6681 raft_consensus.cc:3055] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 09:11:03.063367 6533 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:11:03.063381 6681 raft_consensus.cc:740] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d20fbac1f1243a0b383c9383a1d0a70, State: Running, Role: LEADER
I20251026 09:11:03.063460 6681 consensus_queue.cc:260] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:11:03.063597 6681 raft_consensus.cc:3060] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Advancing to term 4
I20251026 09:11:03.064110 6533 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 4.
I20251026 09:11:03.064196 6681 raft_consensus.cc:2468] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 4.
I20251026 09:11:03.064385 6572 leader_election.cc:304] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed603c643ac44ca9908b171687d7d5c, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:11:03.064495 7669 raft_consensus.cc:2804] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 09:11:03.064637 7669 raft_consensus.cc:697] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:11:03.064718 7669 consensus_queue.cc:237] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } }
I20251026 09:11:03.065261 6428 catalog_manager.cc:5649] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 3 to 4, leader changed from 6d20fbac1f1243a0b383c9383a1d0a70 (127.3.14.3) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 4 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:03.109584 7716 raft_consensus.cc:993] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: : Instructing follower 26c251f040b9472a92191f35891e8eed to start an election
I20251026 09:11:03.109663 7716 raft_consensus.cc:1081] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Signalling peer 26c251f040b9472a92191f35891e8eed to start an election
I20251026 09:11:03.109865 6937 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e"
dest_uuid: "26c251f040b9472a92191f35891e8eed"
from {username='slave'} at 127.0.0.1:53940
I20251026 09:11:03.109961 6937 raft_consensus.cc:493] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:11:03.109984 6937 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Advancing to term 7
I20251026 09:11:03.111961 6937 raft_consensus.cc:515] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:11:03.112102 6937 leader_election.cc:290] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [CANDIDATE]: Term 7 election: Requested vote from peers 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705), 90365b2687854499bddaa63c08c0964f (127.3.14.5:40007)
I20251026 09:11:03.112355 6533 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "26c251f040b9472a92191f35891e8eed" candidate_term: 7 candidate_status { last_received { term: 6 index: 47 } } ignore_live_leader: true dest_uuid: "3ed603c643ac44ca9908b171687d7d5c"
I20251026 09:11:03.112409 7014 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6459af4a4180422a8eccf12fc862499e" candidate_uuid: "26c251f040b9472a92191f35891e8eed" candidate_term: 7 candidate_status { last_received { term: 6 index: 47 } } ignore_live_leader: true dest_uuid: "90365b2687854499bddaa63c08c0964f"
I20251026 09:11:03.112444 6533 raft_consensus.cc:3055] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Stepping down as leader of term 6
I20251026 09:11:03.112470 7014 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 6 FOLLOWER]: Advancing to term 7
I20251026 09:11:03.112478 6533 raft_consensus.cc:740] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3ed603c643ac44ca9908b171687d7d5c, State: Running, Role: LEADER
I20251026 09:11:03.112592 6533 consensus_queue.cc:260] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 6.47, Last appended by leader: 47, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:11:03.112689 6533 raft_consensus.cc:3060] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 6 FOLLOWER]: Advancing to term 7
I20251026 09:11:03.113107 7014 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c251f040b9472a92191f35891e8eed in term 7.
I20251026 09:11:03.113304 6901 leader_election.cc:304] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c251f040b9472a92191f35891e8eed, 90365b2687854499bddaa63c08c0964f; no voters:
I20251026 09:11:03.113305 6533 raft_consensus.cc:2468] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c251f040b9472a92191f35891e8eed in term 7.
I20251026 09:11:03.113418 7693 raft_consensus.cc:2804] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 7 FOLLOWER]: Leader election won for term 7
I20251026 09:11:03.113569 7693 raft_consensus.cc:697] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 7 LEADER]: Becoming Leader. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Running, Role: LEADER
I20251026 09:11:03.113651 7693 consensus_queue.cc:237] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 6.47, Last appended by leader: 47, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } }
I20251026 09:11:03.114210 6428 catalog_manager.cc:5649] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed reported cstate change: term changed from 6 to 7, leader changed from 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1) to 26c251f040b9472a92191f35891e8eed (127.3.14.4). New cstate: current_term: 7 leader_uuid: "26c251f040b9472a92191f35891e8eed" committed_config { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:03.488272 6681 raft_consensus.cc:1275] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 4 index: 39. (index mismatch)
I20251026 09:11:03.488629 7669 consensus_queue.cc:1048] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20fbac1f1243a0b383c9383a1d0a70" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:11:03.490257 6533 raft_consensus.cc:1275] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 4 index: 39. (index mismatch)
I20251026 09:11:03.490540 7731 consensus_queue.cc:1048] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251026 09:11:03.621356 6533 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 7 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 6 index: 47. Preceding OpId from leader: term: 7 index: 48. (index mismatch)
I20251026 09:11:03.621708 7693 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed603c643ac44ca9908b171687d7d5c" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251026 09:11:03.622859 7014 raft_consensus.cc:1275] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 7 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 6 index: 47. Preceding OpId from leader: term: 7 index: 48. (index mismatch)
I20251026 09:11:03.623054 7737 consensus_queue.cc:1048] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "90365b2687854499bddaa63c08c0964f" member_type: VOTER last_known_addr { host: "127.3.14.5" port: 40007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251026 09:11:04.876261 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:04.876754 6605 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "26c251f040b9472a92191f35891e8eed"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:11:04.876850 6605 raft_consensus.cc:606] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Received request to transfer leadership to TS 26c251f040b9472a92191f35891e8eed
I20251026 09:11:04.877077 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 09:11:04.877166 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:05.315400 7669 raft_consensus.cc:993] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 26c251f040b9472a92191f35891e8eed to start an election
I20251026 09:11:05.315500 7669 raft_consensus.cc:1081] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Signalling peer 26c251f040b9472a92191f35891e8eed to start an election
I20251026 09:11:05.315727 6937 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "26c251f040b9472a92191f35891e8eed"
from {username='slave'} at 127.0.0.1:53924
I20251026 09:11:05.315799 6937 raft_consensus.cc:493] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:11:05.315831 6937 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:11:05.316568 6937 raft_consensus.cc:515] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:05.316756 6937 leader_election.cc:290] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [CANDIDATE]: Term 5 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:11:05.316936 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "26c251f040b9472a92191f35891e8eed" candidate_term: 5 candidate_status { last_received { term: 4 index: 44 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:11:05.317005 6605 raft_consensus.cc:3055] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 09:11:05.317029 6605 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:11:05.317102 6605 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 4.44, Last appended by leader: 44, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:05.317191 6605 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:11:05.317953 6605 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c251f040b9472a92191f35891e8eed in term 5.
I20251026 09:11:05.318192 6904 leader_election.cc:304] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c251f040b9472a92191f35891e8eed, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:11:05.318317 7735 raft_consensus.cc:2804] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 5 FOLLOWER]: Leader election won for term 5
I20251026 09:11:05.318449 7735 raft_consensus.cc:697] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 5 LEADER]: Becoming Leader. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Running, Role: LEADER
I20251026 09:11:05.318527 7735 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 4.44, Last appended by leader: 44, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:05.319160 6427 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed reported cstate change: term changed from 4 to 5, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 26c251f040b9472a92191f35891e8eed (127.3.14.4). New cstate: current_term: 5 leader_uuid: "26c251f040b9472a92191f35891e8eed" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:05.320262 7088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "26c251f040b9472a92191f35891e8eed" candidate_term: 5 candidate_status { last_received { term: 4 index: 44 } } ignore_live_leader: true dest_uuid: "485b30c027fe4845b029de91594146fd"
I20251026 09:11:05.320348 7088 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 4 FOLLOWER]: Advancing to term 5
I20251026 09:11:05.321049 7088 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c251f040b9472a92191f35891e8eed in term 5.
I20251026 09:11:05.701551 7088 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 4 index: 44. Preceding OpId from leader: term: 5 index: 45. (index mismatch)
I20251026 09:11:05.701898 7743 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:05.703068 6605 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 4 index: 44. Preceding OpId from leader: term: 5 index: 45. (index mismatch)
I20251026 09:11:05.703421 7743 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251026 09:11:06.877408 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:06.881152 6937 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "26c251f040b9472a92191f35891e8eed"
mode: GRACEFUL
new_leader_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:54212
I20251026 09:11:06.881258 6937 raft_consensus.cc:606] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 5 LEADER]: Received request to transfer leadership to TS cfcab08da3db40e1974902b787ef6361
I20251026 09:11:06.881453 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 09:11:06.881542 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:07.065461 7754 raft_consensus.cc:993] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: : Instructing follower cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:11:07.065582 7754 raft_consensus.cc:1081] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 5 LEADER]: Signalling peer cfcab08da3db40e1974902b787ef6361 to start an election
I20251026 09:11:07.065910 6605 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
from {username='slave'} at 127.0.0.1:44500
I20251026 09:11:07.066020 6605 raft_consensus.cc:493] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:11:07.066059 6605 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Advancing to term 6
I20251026 09:11:07.067042 6605 raft_consensus.cc:515] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:07.067251 6605 leader_election.cc:290] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 6 election: Requested vote from peers 26c251f040b9472a92191f35891e8eed (127.3.14.4:39841), 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:11:07.067478 6937 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: true dest_uuid: "26c251f040b9472a92191f35891e8eed"
I20251026 09:11:07.067484 7088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "cfcab08da3db40e1974902b787ef6361" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: true dest_uuid: "485b30c027fe4845b029de91594146fd"
I20251026 09:11:07.067565 6937 raft_consensus.cc:3055] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 5 LEADER]: Stepping down as leader of term 5
I20251026 09:11:07.067655 7088 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Advancing to term 6
I20251026 09:11:07.067679 6937 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Running, Role: LEADER
I20251026 09:11:07.067819 6937 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 5.45, Last appended by leader: 45, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:07.067907 6937 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 5 FOLLOWER]: Advancing to term 6
I20251026 09:11:07.068301 7088 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 6.
I20251026 09:11:07.068506 6575 leader_election.cc:304] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 485b30c027fe4845b029de91594146fd, cfcab08da3db40e1974902b787ef6361; no voters:
I20251026 09:11:07.068604 7669 raft_consensus.cc:2804] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 09:11:07.068697 6937 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcab08da3db40e1974902b787ef6361 in term 6.
I20251026 09:11:07.068713 7669 raft_consensus.cc:697] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 6 LEADER]: Becoming Leader. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:11:07.068938 7669 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 5.45, Last appended by leader: 45, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:07.069566 6428 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 reported cstate change: term changed from 5 to 6, leader changed from 26c251f040b9472a92191f35891e8eed (127.3.14.4) to cfcab08da3db40e1974902b787ef6361 (127.3.14.2). New cstate: current_term: 6 leader_uuid: "cfcab08da3db40e1974902b787ef6361" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:07.635399 7088 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 6 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 5 index: 45. Preceding OpId from leader: term: 6 index: 46. (index mismatch)
I20251026 09:11:07.635728 7669 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:11:07.636998 6937 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Refusing update from remote peer cfcab08da3db40e1974902b787ef6361: Log matching property violated. Preceding OpId in replica: term: 5 index: 45. Preceding OpId from leader: term: 6 index: 46. (index mismatch)
I20251026 09:11:07.637279 7765 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251026 09:11:08.881742 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:08.882360 6605 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "cfcab08da3db40e1974902b787ef6361"
mode: GRACEFUL
new_leader_uuid: "26c251f040b9472a92191f35891e8eed"
from {username='slave'} at 127.0.0.1:44484
I20251026 09:11:08.882475 6605 raft_consensus.cc:606] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 6 LEADER]: Received request to transfer leadership to TS 26c251f040b9472a92191f35891e8eed
I20251026 09:11:08.882700 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 09:11:08.882776 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:09.133055 7765 raft_consensus.cc:993] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: : Instructing follower 26c251f040b9472a92191f35891e8eed to start an election
I20251026 09:11:09.133181 7765 raft_consensus.cc:1081] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 6 LEADER]: Signalling peer 26c251f040b9472a92191f35891e8eed to start an election
I20251026 09:11:09.133406 6937 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70"
dest_uuid: "26c251f040b9472a92191f35891e8eed"
from {username='slave'} at 127.0.0.1:53924
I20251026 09:11:09.133503 6937 raft_consensus.cc:493] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:11:09.133526 6937 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 6 FOLLOWER]: Advancing to term 7
I20251026 09:11:09.134241 6937 raft_consensus.cc:515] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:09.134385 6937 leader_election.cc:290] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [CANDIDATE]: Term 7 election: Requested vote from peers cfcab08da3db40e1974902b787ef6361 (127.3.14.2:42631), 485b30c027fe4845b029de91594146fd (127.3.14.6:35539)
I20251026 09:11:09.134600 6605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "26c251f040b9472a92191f35891e8eed" candidate_term: 7 candidate_status { last_received { term: 6 index: 46 } } ignore_live_leader: true dest_uuid: "cfcab08da3db40e1974902b787ef6361"
I20251026 09:11:09.134621 7088 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff5ceda9af754029a9b5ac636a17ea70" candidate_uuid: "26c251f040b9472a92191f35891e8eed" candidate_term: 7 candidate_status { last_received { term: 6 index: 46 } } ignore_live_leader: true dest_uuid: "485b30c027fe4845b029de91594146fd"
I20251026 09:11:09.134711 6605 raft_consensus.cc:3055] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 09:11:09.134764 6605 raft_consensus.cc:740] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cfcab08da3db40e1974902b787ef6361, State: Running, Role: LEADER
I20251026 09:11:09.134716 7088 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 6 FOLLOWER]: Advancing to term 7
I20251026 09:11:09.134855 6605 consensus_queue.cc:260] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 6.46, Last appended by leader: 46, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:09.135020 6605 raft_consensus.cc:3060] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Advancing to term 7
I20251026 09:11:09.135483 7088 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c251f040b9472a92191f35891e8eed in term 7.
I20251026 09:11:09.135687 6904 leader_election.cc:304] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c251f040b9472a92191f35891e8eed, 485b30c027fe4845b029de91594146fd; no voters:
I20251026 09:11:09.135808 6605 raft_consensus.cc:2468] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c251f040b9472a92191f35891e8eed in term 7.
I20251026 09:11:09.135823 7754 raft_consensus.cc:2804] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 7 FOLLOWER]: Leader election won for term 7
I20251026 09:11:09.135936 7754 raft_consensus.cc:697] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 7 LEADER]: Becoming Leader. State: Replica: 26c251f040b9472a92191f35891e8eed, State: Running, Role: LEADER
I20251026 09:11:09.136003 7754 consensus_queue.cc:237] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 6.46, Last appended by leader: 46, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } }
I20251026 09:11:09.136560 6427 catalog_manager.cc:5649] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed reported cstate change: term changed from 6 to 7, leader changed from cfcab08da3db40e1974902b787ef6361 (127.3.14.2) to 26c251f040b9472a92191f35891e8eed (127.3.14.4). New cstate: current_term: 7 leader_uuid: "26c251f040b9472a92191f35891e8eed" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c251f040b9472a92191f35891e8eed" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 39841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:09.591487 7088 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 7 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 6 index: 46. Preceding OpId from leader: term: 7 index: 47. (index mismatch)
I20251026 09:11:09.591781 7754 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "485b30c027fe4845b029de91594146fd" member_type: VOTER last_known_addr { host: "127.3.14.6" port: 35539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251026 09:11:09.593276 6604 raft_consensus.cc:1275] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 7 FOLLOWER]: Refusing update from remote peer 26c251f040b9472a92191f35891e8eed: Log matching property violated. Preceding OpId in replica: term: 6 index: 46. Preceding OpId from leader: term: 7 index: 47. (index mismatch)
I20251026 09:11:09.593538 7754 consensus_queue.cc:1048] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcab08da3db40e1974902b787ef6361" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251026 09:11:10.882958 6485 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:10.883610 6937 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ad759f01ba6c4154968854b4b9378651"
dest_uuid: "26c251f040b9472a92191f35891e8eed"
mode: GRACEFUL
new_leader_uuid: "90365b2687854499bddaa63c08c0964f"
from {username='slave'} at 127.0.0.1:54212
I20251026 09:11:10.883709 6937 raft_consensus.cc:606] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 LEADER]: Received request to transfer leadership to TS 90365b2687854499bddaa63c08c0964f
I20251026 09:11:10.883944 6485 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 09:11:10.884021 6485 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:196: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20251026 09:11:10.946456 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:11:10.950241 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:10.950420 3128 tablet_replica.cc:333] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:11:10.950487 3128 raft_consensus.cc:2243] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 7 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.950548 3128 raft_consensus.cc:2272] T 6459af4a4180422a8eccf12fc862499e P 3ed603c643ac44ca9908b171687d7d5c [term 7 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.950752 3128 tablet_replica.cc:333] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:11:10.950837 3128 raft_consensus.cc:2243] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.950901 3128 raft_consensus.cc:2272] T ad759f01ba6c4154968854b4b9378651 P 3ed603c643ac44ca9908b171687d7d5c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.951052 3128 tablet_replica.cc:333] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:11:10.951087 3128 raft_consensus.cc:2243] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.951169 3128 raft_consensus.cc:2272] T 21bd52ca927842ccab72fc70494128d0 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.951310 3128 tablet_replica.cc:333] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:11:10.951395 3128 raft_consensus.cc:2243] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.951436 3128 raft_consensus.cc:2272] T e65f6e7109bc4eda8d83d836786c4d16 P 3ed603c643ac44ca9908b171687d7d5c [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.951591 3128 tablet_replica.cc:333] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:11:10.951624 3128 raft_consensus.cc:2243] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 5 LEADER]: Raft consensus shutting down.
I20251026 09:11:10.951696 3128 raft_consensus.cc:2272] T ac1d836cf54d42c39eb2d8279fd17f90 P 3ed603c643ac44ca9908b171687d7d5c [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.951828 3128 tablet_replica.cc:333] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c: stopping tablet replica
I20251026 09:11:10.951874 3128 raft_consensus.cc:2243] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 6 LEADER]: Raft consensus shutting down.
I20251026 09:11:10.951930 3128 raft_consensus.cc:2272] T 90072194423a4007b85cbc3f31889400 P 3ed603c643ac44ca9908b171687d7d5c [term 6 FOLLOWER]: Raft consensus is shut down!
W20251026 09:11:10.953716 6901 connection.cc:537] client connection to 127.3.14.1:33705 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251026 09:11:10.953796 6901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
W20251026 09:11:10.954272 6901 consensus_peers.cc:597] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed -> Peer 3ed603c643ac44ca9908b171687d7d5c (127.3.14.1:33705): Couldn't send request to peer 3ed603c643ac44ca9908b171687d7d5c. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:33705: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:10.965958 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:11:10.967765 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:11:10.971779 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:10.971932 3128 tablet_replica.cc:333] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:11:10.972005 3128 raft_consensus.cc:2243] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 4 LEADER]: Raft consensus shutting down.
I20251026 09:11:10.972083 3128 raft_consensus.cc:2272] T e65f6e7109bc4eda8d83d836786c4d16 P cfcab08da3db40e1974902b787ef6361 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.972316 3128 tablet_replica.cc:333] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:11:10.972358 3128 raft_consensus.cc:2243] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 7 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.972390 3128 raft_consensus.cc:2272] T ff5ceda9af754029a9b5ac636a17ea70 P cfcab08da3db40e1974902b787ef6361 [term 7 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.972477 3128 tablet_replica.cc:333] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:11:10.972505 3128 raft_consensus.cc:2243] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 LEADER]: Raft consensus shutting down.
I20251026 09:11:10.972544 3128 raft_consensus.cc:2272] T 22a8d92b27b14f89a4728a8f50db932b P cfcab08da3db40e1974902b787ef6361 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.972631 3128 tablet_replica.cc:333] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:11:10.972658 3128 raft_consensus.cc:2243] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.972679 3128 raft_consensus.cc:2272] T ac1d836cf54d42c39eb2d8279fd17f90 P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.972805 3128 tablet_replica.cc:333] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:11:10.972846 3128 raft_consensus.cc:2243] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.972878 3128 raft_consensus.cc:2272] T f9723f81063e4f23bb2300f2c7bd50eb P cfcab08da3db40e1974902b787ef6361 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.973026 3128 tablet_replica.cc:333] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361: stopping tablet replica
I20251026 09:11:10.973101 3128 raft_consensus.cc:2243] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.973167 3128 raft_consensus.cc:2272] T 90072194423a4007b85cbc3f31889400 P cfcab08da3db40e1974902b787ef6361 [term 6 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.985879 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:11:10.987885 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:11:10.992552 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:10.992720 3128 tablet_replica.cc:333] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:11:10.992799 3128 raft_consensus.cc:2243] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.992934 3128 raft_consensus.cc:2272] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.993167 3128 tablet_replica.cc:333] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:11:10.993223 3128 raft_consensus.cc:2243] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.993269 3128 raft_consensus.cc:2272] T e65f6e7109bc4eda8d83d836786c4d16 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.993479 3128 tablet_replica.cc:333] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:11:10.993517 3128 raft_consensus.cc:2243] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Raft consensus shutting down.
I20251026 09:11:10.993587 3128 raft_consensus.cc:2272] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.993728 3128 tablet_replica.cc:333] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:11:10.993767 3128 raft_consensus.cc:2243] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 LEADER]: Raft consensus shutting down.
I20251026 09:11:10.993818 3128 raft_consensus.cc:2272] T e002d8e9fb004210be27d87c5a23f9c3 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.993947 3128 tablet_replica.cc:333] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:11:10.993983 3128 raft_consensus.cc:2243] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.994025 3128 raft_consensus.cc:2272] T f9723f81063e4f23bb2300f2c7bd50eb P 6d20fbac1f1243a0b383c9383a1d0a70 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:10.994163 3128 tablet_replica.cc:333] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70: stopping tablet replica
I20251026 09:11:10.994197 3128 raft_consensus.cc:2243] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:10.994239 3128 raft_consensus.cc:2272] T 90072194423a4007b85cbc3f31889400 P 6d20fbac1f1243a0b383c9383a1d0a70 [term 6 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.007416 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:11:11.009308 3128 tablet_server.cc:178] TabletServer@127.3.14.4:0 shutting down...
I20251026 09:11:11.012849 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:11.013077 3128 tablet_replica.cc:333] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed: stopping tablet replica
I20251026 09:11:11.013150 3128 raft_consensus.cc:2243] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.013253 3128 raft_consensus.cc:2272] T 22a8d92b27b14f89a4728a8f50db932b P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.013484 3128 tablet_replica.cc:333] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed: stopping tablet replica
I20251026 09:11:11.013535 3128 raft_consensus.cc:2243] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 7 LEADER]: Raft consensus shutting down.
I20251026 09:11:11.013587 3128 raft_consensus.cc:2272] T 6459af4a4180422a8eccf12fc862499e P 26c251f040b9472a92191f35891e8eed [term 7 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.013738 3128 tablet_replica.cc:333] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed: stopping tablet replica
I20251026 09:11:11.013782 3128 raft_consensus.cc:2243] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.013826 3128 raft_consensus.cc:2272] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.013952 3128 tablet_replica.cc:333] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed: stopping tablet replica
I20251026 09:11:11.013996 3128 raft_consensus.cc:2243] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.014068 3128 raft_consensus.cc:2272] T e002d8e9fb004210be27d87c5a23f9c3 P 26c251f040b9472a92191f35891e8eed [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.014189 3128 tablet_replica.cc:333] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed: stopping tablet replica
I20251026 09:11:11.014221 3128 raft_consensus.cc:2243] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 LEADER]: Raft consensus shutting down.
I20251026 09:11:11.014276 3128 raft_consensus.cc:2272] T ad759f01ba6c4154968854b4b9378651 P 26c251f040b9472a92191f35891e8eed [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.014417 3128 tablet_replica.cc:333] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed: stopping tablet replica
I20251026 09:11:11.014449 3128 raft_consensus.cc:2243] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 7 LEADER]: Raft consensus shutting down.
I20251026 09:11:11.014500 3128 raft_consensus.cc:2272] T ff5ceda9af754029a9b5ac636a17ea70 P 26c251f040b9472a92191f35891e8eed [term 7 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.026943 3128 tablet_server.cc:195] TabletServer@127.3.14.4:0 shutdown complete.
I20251026 09:11:11.028554 3128 tablet_server.cc:178] TabletServer@127.3.14.5:0 shutting down...
I20251026 09:11:11.032503 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:11.032711 3128 tablet_replica.cc:333] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f: stopping tablet replica
I20251026 09:11:11.032794 3128 raft_consensus.cc:2243] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.032871 3128 raft_consensus.cc:2272] T 22a8d92b27b14f89a4728a8f50db932b P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.033079 3128 tablet_replica.cc:333] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f: stopping tablet replica
I20251026 09:11:11.033138 3128 raft_consensus.cc:2243] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.033198 3128 raft_consensus.cc:2272] T e002d8e9fb004210be27d87c5a23f9c3 P 90365b2687854499bddaa63c08c0964f [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.033334 3128 tablet_replica.cc:333] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f: stopping tablet replica
I20251026 09:11:11.033386 3128 raft_consensus.cc:2243] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 5 LEADER]: Raft consensus shutting down.
I20251026 09:11:11.033457 3128 raft_consensus.cc:2272] T 21bd52ca927842ccab72fc70494128d0 P 90365b2687854499bddaa63c08c0964f [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.033586 3128 tablet_replica.cc:333] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f: stopping tablet replica
I20251026 09:11:11.033627 3128 raft_consensus.cc:2243] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.033703 3128 raft_consensus.cc:2272] T ad759f01ba6c4154968854b4b9378651 P 90365b2687854499bddaa63c08c0964f [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.033833 3128 tablet_replica.cc:333] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f: stopping tablet replica
I20251026 09:11:11.033876 3128 raft_consensus.cc:2243] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.033929 3128 raft_consensus.cc:2272] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 90365b2687854499bddaa63c08c0964f [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.034063 3128 tablet_replica.cc:333] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f: stopping tablet replica
I20251026 09:11:11.034109 3128 raft_consensus.cc:2243] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 7 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.034155 3128 raft_consensus.cc:2272] T 6459af4a4180422a8eccf12fc862499e P 90365b2687854499bddaa63c08c0964f [term 7 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.046831 3128 tablet_server.cc:195] TabletServer@127.3.14.5:0 shutdown complete.
I20251026 09:11:11.048592 3128 tablet_server.cc:178] TabletServer@127.3.14.6:0 shutting down...
I20251026 09:11:11.052143 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:11.052321 3128 tablet_replica.cc:333] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd: stopping tablet replica
I20251026 09:11:11.052430 3128 raft_consensus.cc:2243] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.052505 3128 raft_consensus.cc:2272] T 21bd52ca927842ccab72fc70494128d0 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.052690 3128 tablet_replica.cc:333] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd: stopping tablet replica
I20251026 09:11:11.052727 3128 raft_consensus.cc:2243] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 7 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.052775 3128 raft_consensus.cc:2272] T ff5ceda9af754029a9b5ac636a17ea70 P 485b30c027fe4845b029de91594146fd [term 7 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.052873 3128 tablet_replica.cc:333] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd: stopping tablet replica
I20251026 09:11:11.052906 3128 raft_consensus.cc:2243] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 5 LEADER]: Raft consensus shutting down.
I20251026 09:11:11.052949 3128 raft_consensus.cc:2272] T f9723f81063e4f23bb2300f2c7bd50eb P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.053038 3128 tablet_replica.cc:333] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd: stopping tablet replica
I20251026 09:11:11.053068 3128 raft_consensus.cc:2243] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.053122 3128 raft_consensus.cc:2272] T 4bb5819b36e244d4b0ab2c0e110e51e9 P 485b30c027fe4845b029de91594146fd [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.053218 3128 tablet_replica.cc:333] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd: stopping tablet replica
I20251026 09:11:11.053248 3128 raft_consensus.cc:2243] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 5 LEADER]: Raft consensus shutting down.
I20251026 09:11:11.053296 3128 raft_consensus.cc:2272] T 75fb5eb7df3b4c87bfdaa0b3f0efe601 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.053395 3128 tablet_replica.cc:333] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd: stopping tablet replica
I20251026 09:11:11.053427 3128 raft_consensus.cc:2243] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:11.053452 3128 raft_consensus.cc:2272] T ac1d836cf54d42c39eb2d8279fd17f90 P 485b30c027fe4845b029de91594146fd [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.065562 3128 tablet_server.cc:195] TabletServer@127.3.14.6:0 shutdown complete.
I20251026 09:11:11.067209 3128 master.cc:561] Master@127.3.14.62:38937 shutting down...
I20251026 09:11:11.070243 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:11.070345 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:11.070366 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0194cb86687941f2a46146ccf9f70022: stopping tablet replica
I20251026 09:11:11.083089 3128 master.cc:583] Master@127.3.14.62:38937 shutdown complete.
I20251026 09:11:11.085089 3128 test_util.cc:183] -----------------------------------------------
I20251026 09:11:11.085173 3128 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761469742341867-3128-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (36284 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20251026 09:11:11.085753 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:37769
I20251026 09:11:11.085917 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:11:11.087626 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:11.087656 7791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:11.087818 7788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:11.087656 7789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:11.087968 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:11.088006 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:11.088023 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469871088024 us; error 0 us; skew 500 ppm
I20251026 09:11:11.088668 3128 webserver.cc:492] Webserver started at http://127.3.14.62:37227/ using document root <none> and password file <none>
I20251026 09:11:11.088755 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:11.088798 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:11.088836 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:11.089103 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "65745862afe44f4d9f82934393ef6f60"
format_stamp: "Formatted at 2025-10-26 09:11:11 on dist-test-slave-w8k0"
I20251026 09:11:11.089998 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 09:11:11.090456 7796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.090595 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.090646 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "65745862afe44f4d9f82934393ef6f60"
format_stamp: "Formatted at 2025-10-26 09:11:11 on dist-test-slave-w8k0"
I20251026 09:11:11.090694 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:11.110507 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:11.110734 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:11.113619 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:37769
I20251026 09:11:11.114665 7848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:37769 every 8 connection(s)
I20251026 09:11:11.120894 7849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:11.122153 7849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60: Bootstrap starting.
I20251026 09:11:11.122504 7849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:11.123095 7849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60: No bootstrap required, opened a new log
I20251026 09:11:11.123256 7849 raft_consensus.cc:359] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65745862afe44f4d9f82934393ef6f60" member_type: VOTER }
I20251026 09:11:11.123315 7849 raft_consensus.cc:385] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:11.123378 7849 raft_consensus.cc:740] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65745862afe44f4d9f82934393ef6f60, State: Initialized, Role: FOLLOWER
I20251026 09:11:11.123446 7849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65745862afe44f4d9f82934393ef6f60" member_type: VOTER }
I20251026 09:11:11.123493 7849 raft_consensus.cc:399] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:11:11.123522 7849 raft_consensus.cc:493] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:11:11.123553 7849 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:11.124154 7849 raft_consensus.cc:515] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65745862afe44f4d9f82934393ef6f60" member_type: VOTER }
I20251026 09:11:11.124223 7849 leader_election.cc:304] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65745862afe44f4d9f82934393ef6f60; no voters:
I20251026 09:11:11.124372 7849 leader_election.cc:290] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:11:11.124440 7852 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:11.124533 7849 sys_catalog.cc:565] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:11:11.124644 7852 raft_consensus.cc:697] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 1 LEADER]: Becoming Leader. State: Replica: 65745862afe44f4d9f82934393ef6f60, State: Running, Role: LEADER
I20251026 09:11:11.124711 7852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65745862afe44f4d9f82934393ef6f60" member_type: VOTER }
I20251026 09:11:11.124991 7853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65745862afe44f4d9f82934393ef6f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65745862afe44f4d9f82934393ef6f60" member_type: VOTER } }
I20251026 09:11:11.124997 7854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65745862afe44f4d9f82934393ef6f60. Latest consensus state: current_term: 1 leader_uuid: "65745862afe44f4d9f82934393ef6f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65745862afe44f4d9f82934393ef6f60" member_type: VOTER } }
I20251026 09:11:11.125165 7853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:11.125180 7854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:11.125530 7862 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:11:11.125684 7862 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:11:11.125957 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:11:11.126313 7862 catalog_manager.cc:1357] Generated new cluster ID: 787a99efa0a4463e8dea76c58308bed1
I20251026 09:11:11.126360 7862 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:11:11.136232 7862 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:11:11.136723 7862 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:11:11.142454 7862 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60: Generated new TSK 0
I20251026 09:11:11.142562 7862 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:11:11.157974 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:11.159584 7870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:11.159632 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:11.159701 7873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:11.159783 7871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:11.159937 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:11.159979 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:11.159994 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469871159995 us; error 0 us; skew 500 ppm
I20251026 09:11:11.160810 3128 webserver.cc:492] Webserver started at http://127.3.14.1:38455/ using document root <none> and password file <none>
I20251026 09:11:11.160908 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:11.160956 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:11.161007 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:11.161307 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "5a8e2c3d243249e295b25cf18e0d8d58"
format_stamp: "Formatted at 2025-10-26 09:11:11 on dist-test-slave-w8k0"
I20251026 09:11:11.162339 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:11.162884 7878 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.163069 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:11.163125 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "5a8e2c3d243249e295b25cf18e0d8d58"
format_stamp: "Formatted at 2025-10-26 09:11:11 on dist-test-slave-w8k0"
I20251026 09:11:11.163187 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:11.177803 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:11.178082 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:11.178303 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:11.178545 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:11.178579 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.178617 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:11.178648 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.181732 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:38763
I20251026 09:11:11.182291 7941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:38763 every 8 connection(s)
I20251026 09:11:11.182348 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:11.183876 7949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:11.183950 7945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:11.184163 7947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:11.184264 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:11.184379 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:11.184417 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:11.184437 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469871184437 us; error 0 us; skew 500 ppm
I20251026 09:11:11.185257 3128 webserver.cc:492] Webserver started at http://127.3.14.2:43941/ using document root <none> and password file <none>
I20251026 09:11:11.185374 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:11.185433 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:11.185490 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:11.185808 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "5ea212a715184edab5c65be2f1e7b542"
format_stamp: "Formatted at 2025-10-26 09:11:11 on dist-test-slave-w8k0"
I20251026 09:11:11.186762 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:11.187304 7954 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.187491 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.187538 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "5ea212a715184edab5c65be2f1e7b542"
format_stamp: "Formatted at 2025-10-26 09:11:11 on dist-test-slave-w8k0"
I20251026 09:11:11.187578 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:11.189294 7942 heartbeater.cc:344] Connected to a master server at 127.3.14.62:37769
I20251026 09:11:11.189359 7942 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:11.189493 7942 heartbeater.cc:507] Master 127.3.14.62:37769 requested a full tablet report, sending...
I20251026 09:11:11.189736 7813 ts_manager.cc:194] Registered new tserver with Master: 5a8e2c3d243249e295b25cf18e0d8d58 (127.3.14.1:38763)
I20251026 09:11:11.190311 7813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54110
I20251026 09:11:11.217568 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:11.217796 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:11.217965 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:11.218189 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:11.218221 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.218258 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:11.218286 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.221222 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:43439
I20251026 09:11:11.221323 8017 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:43439 every 8 connection(s)
I20251026 09:11:11.221707 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:11.223508 8022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:11.223508 8021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:11.223711 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:11.223613 8024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:11.223995 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:11.224030 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:11.224056 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469871224054 us; error 0 us; skew 500 ppm
I20251026 09:11:11.224825 3128 webserver.cc:492] Webserver started at http://127.3.14.3:34157/ using document root <none> and password file <none>
I20251026 09:11:11.224922 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:11.224982 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:11.225039 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:11.225426 8018 heartbeater.cc:344] Connected to a master server at 127.3.14.62:37769
I20251026 09:11:11.225498 8018 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:11.225509 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "e6659534443548dca10d69a62a2e0fa1"
format_stamp: "Formatted at 2025-10-26 09:11:11 on dist-test-slave-w8k0"
I20251026 09:11:11.225625 8018 heartbeater.cc:507] Master 127.3.14.62:37769 requested a full tablet report, sending...
I20251026 09:11:11.225888 7813 ts_manager.cc:194] Registered new tserver with Master: 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2:43439)
I20251026 09:11:11.226425 7813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54126
I20251026 09:11:11.226856 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 09:11:11.227473 8029 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.227617 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.227664 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "e6659534443548dca10d69a62a2e0fa1"
format_stamp: "Formatted at 2025-10-26 09:11:11 on dist-test-slave-w8k0"
I20251026 09:11:11.227720 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:11.240923 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:11.241179 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:11.241358 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:11.241580 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:11.241614 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.241652 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:11.241679 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:11.244788 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:45531
I20251026 09:11:11.244854 8092 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:45531 every 8 connection(s)
I20251026 09:11:11.248102 8093 heartbeater.cc:344] Connected to a master server at 127.3.14.62:37769
I20251026 09:11:11.248190 8093 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:11.248301 8093 heartbeater.cc:507] Master 127.3.14.62:37769 requested a full tablet report, sending...
I20251026 09:11:11.248716 7813 ts_manager.cc:194] Registered new tserver with Master: e6659534443548dca10d69a62a2e0fa1 (127.3.14.3:45531)
I20251026 09:11:11.249099 7813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54130
I20251026 09:11:11.249423 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004436866s
I20251026 09:11:12.190892 7942 heartbeater.cc:499] Master 127.3.14.62:37769 was elected leader, sending a full tablet report...
I20251026 09:11:12.227075 8018 heartbeater.cc:499] Master 127.3.14.62:37769 was elected leader, sending a full tablet report...
I20251026 09:11:12.249758 8093 heartbeater.cc:499] Master 127.3.14.62:37769 was elected leader, sending a full tablet report...
I20251026 09:11:12.275233 3128 test_util.cc:276] Using random seed: 704970525
I20251026 09:11:12.281138 7813 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54134:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:11:12.281369 7813 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:11:12.286973 8057 tablet_service.cc:1505] Processing CreateTablet for tablet e27b2e91f46f460a98311ae73d00c61c (DEFAULT_TABLE table=test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251026 09:11:12.286990 8055 tablet_service.cc:1505] Processing CreateTablet for tablet 37609a723dc649ccb93995d750aa7854 (DEFAULT_TABLE table=test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251026 09:11:12.286973 8056 tablet_service.cc:1505] Processing CreateTablet for tablet 90e2755d45eb496f89852bdb34bb421b (DEFAULT_TABLE table=test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251026 09:11:12.287213 8057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27b2e91f46f460a98311ae73d00c61c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:12.287272 8056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90e2755d45eb496f89852bdb34bb421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:12.287559 8055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37609a723dc649ccb93995d750aa7854. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:12.289018 7982 tablet_service.cc:1505] Processing CreateTablet for tablet e27b2e91f46f460a98311ae73d00c61c (DEFAULT_TABLE table=test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251026 09:11:12.289150 7982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27b2e91f46f460a98311ae73d00c61c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:12.289172 7904 tablet_service.cc:1505] Processing CreateTablet for tablet 37609a723dc649ccb93995d750aa7854 (DEFAULT_TABLE table=test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251026 09:11:12.289275 7904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37609a723dc649ccb93995d750aa7854. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:12.289487 8113 tablet_bootstrap.cc:492] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1: Bootstrap starting.
I20251026 09:11:12.289878 8113 tablet_bootstrap.cc:654] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:12.290488 7905 tablet_service.cc:1505] Processing CreateTablet for tablet 90e2755d45eb496f89852bdb34bb421b (DEFAULT_TABLE table=test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251026 09:11:12.290525 7906 tablet_service.cc:1505] Processing CreateTablet for tablet e27b2e91f46f460a98311ae73d00c61c (DEFAULT_TABLE table=test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251026 09:11:12.290607 7905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90e2755d45eb496f89852bdb34bb421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:12.290675 8116 tablet_bootstrap.cc:492] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542: Bootstrap starting.
I20251026 09:11:12.290733 7981 tablet_service.cc:1505] Processing CreateTablet for tablet 90e2755d45eb496f89852bdb34bb421b (DEFAULT_TABLE table=test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251026 09:11:12.290808 7981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90e2755d45eb496f89852bdb34bb421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:12.291152 8116 tablet_bootstrap.cc:654] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:12.290689 7980 tablet_service.cc:1505] Processing CreateTablet for tablet 37609a723dc649ccb93995d750aa7854 (DEFAULT_TABLE table=test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251026 09:11:12.292210 7980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37609a723dc649ccb93995d750aa7854. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:12.292526 8113 tablet_bootstrap.cc:492] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1: No bootstrap required, opened a new log
I20251026 09:11:12.292562 8113 ts_tablet_manager.cc:1403] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251026 09:11:12.292693 8113 raft_consensus.cc:359] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.292743 8113 raft_consensus.cc:385] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:12.292757 8113 raft_consensus.cc:740] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6659534443548dca10d69a62a2e0fa1, State: Initialized, Role: FOLLOWER
I20251026 09:11:12.292800 8113 consensus_queue.cc:260] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.292922 8113 ts_tablet_manager.cc:1434] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:12.290468 8115 tablet_bootstrap.cc:492] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58: Bootstrap starting.
I20251026 09:11:12.293320 7906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27b2e91f46f460a98311ae73d00c61c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:12.293404 8113 tablet_bootstrap.cc:492] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1: Bootstrap starting.
I20251026 09:11:12.293833 8113 tablet_bootstrap.cc:654] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:12.294239 8115 tablet_bootstrap.cc:654] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:12.294428 8113 tablet_bootstrap.cc:492] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1: No bootstrap required, opened a new log
I20251026 09:11:12.294473 8113 ts_tablet_manager.cc:1403] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:12.294652 8113 raft_consensus.cc:359] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.294716 8113 raft_consensus.cc:385] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:12.294737 8113 raft_consensus.cc:740] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6659534443548dca10d69a62a2e0fa1, State: Initialized, Role: FOLLOWER
I20251026 09:11:12.294790 8115 tablet_bootstrap.cc:492] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58: No bootstrap required, opened a new log
I20251026 09:11:12.294799 8113 consensus_queue.cc:260] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.294853 8115 ts_tablet_manager.cc:1403] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251026 09:11:12.294865 8116 tablet_bootstrap.cc:492] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542: No bootstrap required, opened a new log
I20251026 09:11:12.294922 8116 ts_tablet_manager.cc:1403] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20251026 09:11:12.294942 8113 ts_tablet_manager.cc:1434] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.294997 8113 tablet_bootstrap.cc:492] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1: Bootstrap starting.
I20251026 09:11:12.295025 8115 raft_consensus.cc:359] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.295055 8116 raft_consensus.cc:359] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.295083 8115 raft_consensus.cc:385] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:12.295102 8116 raft_consensus.cc:385] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:12.295118 8115 raft_consensus.cc:740] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8e2c3d243249e295b25cf18e0d8d58, State: Initialized, Role: FOLLOWER
I20251026 09:11:12.295125 8116 raft_consensus.cc:740] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea212a715184edab5c65be2f1e7b542, State: Initialized, Role: FOLLOWER
I20251026 09:11:12.295189 8115 consensus_queue.cc:260] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.295213 8116 consensus_queue.cc:260] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.295379 8116 ts_tablet_manager.cc:1434] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.295380 8115 ts_tablet_manager.cc:1434] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:12.295487 8113 tablet_bootstrap.cc:654] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:12.295754 8115 tablet_bootstrap.cc:492] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58: Bootstrap starting.
I20251026 09:11:12.296234 8113 tablet_bootstrap.cc:492] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1: No bootstrap required, opened a new log
I20251026 09:11:12.296289 8113 ts_tablet_manager.cc:1403] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:12.296455 8113 raft_consensus.cc:359] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.296509 8113 raft_consensus.cc:385] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:12.296531 8113 raft_consensus.cc:740] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6659534443548dca10d69a62a2e0fa1, State: Initialized, Role: FOLLOWER
I20251026 09:11:12.296540 8115 tablet_bootstrap.cc:654] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:12.296583 8113 consensus_queue.cc:260] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.296672 8113 ts_tablet_manager.cc:1434] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.297056 8116 tablet_bootstrap.cc:492] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542: Bootstrap starting.
I20251026 09:11:12.297298 8115 tablet_bootstrap.cc:492] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58: No bootstrap required, opened a new log
I20251026 09:11:12.297454 8115 ts_tablet_manager.cc:1403] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:12.297600 8116 tablet_bootstrap.cc:654] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:12.297645 8115 raft_consensus.cc:359] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.297722 8115 raft_consensus.cc:385] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:12.297742 8115 raft_consensus.cc:740] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8e2c3d243249e295b25cf18e0d8d58, State: Initialized, Role: FOLLOWER
I20251026 09:11:12.297801 8115 consensus_queue.cc:260] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.297894 8115 ts_tablet_manager.cc:1434] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.297991 8115 tablet_bootstrap.cc:492] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58: Bootstrap starting.
I20251026 09:11:12.298449 8116 tablet_bootstrap.cc:492] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542: No bootstrap required, opened a new log
I20251026 09:11:12.298514 8116 ts_tablet_manager.cc:1403] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:12.298652 8115 tablet_bootstrap.cc:654] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:12.298683 8116 raft_consensus.cc:359] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.298768 8116 raft_consensus.cc:385] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:12.298791 8116 raft_consensus.cc:740] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea212a715184edab5c65be2f1e7b542, State: Initialized, Role: FOLLOWER
I20251026 09:11:12.298841 8116 consensus_queue.cc:260] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.298923 8116 ts_tablet_manager.cc:1434] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.298974 8116 tablet_bootstrap.cc:492] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542: Bootstrap starting.
I20251026 09:11:12.299202 8115 tablet_bootstrap.cc:492] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58: No bootstrap required, opened a new log
I20251026 09:11:12.299237 8115 ts_tablet_manager.cc:1403] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:12.299322 8116 tablet_bootstrap.cc:654] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:12.299403 8115 raft_consensus.cc:359] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.299458 8115 raft_consensus.cc:385] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:12.299485 8115 raft_consensus.cc:740] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8e2c3d243249e295b25cf18e0d8d58, State: Initialized, Role: FOLLOWER
I20251026 09:11:12.299557 8115 consensus_queue.cc:260] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.299822 8115 ts_tablet_manager.cc:1434] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:12.299944 8116 tablet_bootstrap.cc:492] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542: No bootstrap required, opened a new log
I20251026 09:11:12.299996 8116 ts_tablet_manager.cc:1403] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:12.300134 8116 raft_consensus.cc:359] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.300190 8116 raft_consensus.cc:385] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:12.300215 8116 raft_consensus.cc:740] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea212a715184edab5c65be2f1e7b542, State: Initialized, Role: FOLLOWER
I20251026 09:11:12.300283 8116 consensus_queue.cc:260] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.300345 8122 raft_consensus.cc:493] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:12.300398 8122 raft_consensus.cc:515] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.300447 8116 ts_tablet_manager.cc:1434] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.300849 8122 leader_election.cc:290] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a8e2c3d243249e295b25cf18e0d8d58 (127.3.14.1:38763), e6659534443548dca10d69a62a2e0fa1 (127.3.14.3:45531)
I20251026 09:11:12.303514 7916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e27b2e91f46f460a98311ae73d00c61c" candidate_uuid: "5ea212a715184edab5c65be2f1e7b542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" is_pre_election: true
I20251026 09:11:12.303563 8067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e27b2e91f46f460a98311ae73d00c61c" candidate_uuid: "5ea212a715184edab5c65be2f1e7b542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6659534443548dca10d69a62a2e0fa1" is_pre_election: true
I20251026 09:11:12.303617 7916 raft_consensus.cc:2468] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea212a715184edab5c65be2f1e7b542 in term 0.
I20251026 09:11:12.303645 8067 raft_consensus.cc:2468] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea212a715184edab5c65be2f1e7b542 in term 0.
I20251026 09:11:12.303762 7958 leader_election.cc:304] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a8e2c3d243249e295b25cf18e0d8d58, 5ea212a715184edab5c65be2f1e7b542; no voters:
I20251026 09:11:12.303881 8122 raft_consensus.cc:2804] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:12.303927 8122 raft_consensus.cc:493] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:12.303944 8122 raft_consensus.cc:3060] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:12.304631 8122 raft_consensus.cc:515] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.304773 8122 leader_election.cc:290] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [CANDIDATE]: Term 1 election: Requested vote from peers 5a8e2c3d243249e295b25cf18e0d8d58 (127.3.14.1:38763), e6659534443548dca10d69a62a2e0fa1 (127.3.14.3:45531)
I20251026 09:11:12.304898 7916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e27b2e91f46f460a98311ae73d00c61c" candidate_uuid: "5ea212a715184edab5c65be2f1e7b542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8e2c3d243249e295b25cf18e0d8d58"
I20251026 09:11:12.304906 8067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e27b2e91f46f460a98311ae73d00c61c" candidate_uuid: "5ea212a715184edab5c65be2f1e7b542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6659534443548dca10d69a62a2e0fa1"
I20251026 09:11:12.304979 7916 raft_consensus.cc:3060] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:12.304984 8067 raft_consensus.cc:3060] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:12.305728 7916 raft_consensus.cc:2468] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea212a715184edab5c65be2f1e7b542 in term 1.
I20251026 09:11:12.305728 8067 raft_consensus.cc:2468] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea212a715184edab5c65be2f1e7b542 in term 1.
I20251026 09:11:12.305924 7958 leader_election.cc:304] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5a8e2c3d243249e295b25cf18e0d8d58, 5ea212a715184edab5c65be2f1e7b542, e6659534443548dca10d69a62a2e0fa1; no voters:
I20251026 09:11:12.306034 8122 raft_consensus.cc:2804] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:12.306098 8122 raft_consensus.cc:697] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Becoming Leader. State: Replica: 5ea212a715184edab5c65be2f1e7b542, State: Running, Role: LEADER
I20251026 09:11:12.306162 8122 consensus_queue.cc:237] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.306748 7813 catalog_manager.cc:5649] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "5ea212a715184edab5c65be2f1e7b542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:12.328040 8118 raft_consensus.cc:493] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:12.328192 8118 raft_consensus.cc:515] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.328433 8118 leader_election.cc:290] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a8e2c3d243249e295b25cf18e0d8d58 (127.3.14.1:38763), 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2:43439)
I20251026 09:11:12.331456 7916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37609a723dc649ccb93995d750aa7854" candidate_uuid: "e6659534443548dca10d69a62a2e0fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" is_pre_election: true
I20251026 09:11:12.331487 7992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37609a723dc649ccb93995d750aa7854" candidate_uuid: "e6659534443548dca10d69a62a2e0fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea212a715184edab5c65be2f1e7b542" is_pre_election: true
I20251026 09:11:12.331576 7916 raft_consensus.cc:2468] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6659534443548dca10d69a62a2e0fa1 in term 0.
I20251026 09:11:12.331595 7992 raft_consensus.cc:2468] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6659534443548dca10d69a62a2e0fa1 in term 0.
I20251026 09:11:12.331761 8033 leader_election.cc:304] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea212a715184edab5c65be2f1e7b542, e6659534443548dca10d69a62a2e0fa1; no voters:
I20251026 09:11:12.331875 8118 raft_consensus.cc:2804] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:12.331925 8118 raft_consensus.cc:493] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:12.331947 8118 raft_consensus.cc:3060] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:12.332648 8118 raft_consensus.cc:515] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.332782 8118 leader_election.cc:290] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [CANDIDATE]: Term 1 election: Requested vote from peers 5a8e2c3d243249e295b25cf18e0d8d58 (127.3.14.1:38763), 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2:43439)
I20251026 09:11:12.332959 7916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37609a723dc649ccb93995d750aa7854" candidate_uuid: "e6659534443548dca10d69a62a2e0fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8e2c3d243249e295b25cf18e0d8d58"
I20251026 09:11:12.333014 7916 raft_consensus.cc:3060] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:12.332983 7992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37609a723dc649ccb93995d750aa7854" candidate_uuid: "e6659534443548dca10d69a62a2e0fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea212a715184edab5c65be2f1e7b542"
I20251026 09:11:12.333052 7992 raft_consensus.cc:3060] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:12.333653 7916 raft_consensus.cc:2468] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6659534443548dca10d69a62a2e0fa1 in term 1.
I20251026 09:11:12.333678 7992 raft_consensus.cc:2468] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6659534443548dca10d69a62a2e0fa1 in term 1.
I20251026 09:11:12.333832 8033 leader_election.cc:304] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea212a715184edab5c65be2f1e7b542, e6659534443548dca10d69a62a2e0fa1; no voters:
I20251026 09:11:12.333930 8118 raft_consensus.cc:2804] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:12.334118 8118 raft_consensus.cc:697] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 1 LEADER]: Becoming Leader. State: Replica: e6659534443548dca10d69a62a2e0fa1, State: Running, Role: LEADER
I20251026 09:11:12.334179 8118 consensus_queue.cc:237] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.334786 7813 catalog_manager.cc:5649] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6659534443548dca10d69a62a2e0fa1 (127.3.14.3). New cstate: current_term: 1 leader_uuid: "e6659534443548dca10d69a62a2e0fa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } health_report { overall_health: HEALTHY } } }
I20251026 09:11:12.341009 8122 raft_consensus.cc:493] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:12.341127 8122 raft_consensus.cc:515] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.341302 8122 leader_election.cc:290] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a8e2c3d243249e295b25cf18e0d8d58 (127.3.14.1:38763), e6659534443548dca10d69a62a2e0fa1 (127.3.14.3:45531)
I20251026 09:11:12.341549 8067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90e2755d45eb496f89852bdb34bb421b" candidate_uuid: "5ea212a715184edab5c65be2f1e7b542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6659534443548dca10d69a62a2e0fa1" is_pre_election: true
I20251026 09:11:12.341562 7916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90e2755d45eb496f89852bdb34bb421b" candidate_uuid: "5ea212a715184edab5c65be2f1e7b542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" is_pre_election: true
I20251026 09:11:12.341688 8067 raft_consensus.cc:2468] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea212a715184edab5c65be2f1e7b542 in term 0.
I20251026 09:11:12.341706 7916 raft_consensus.cc:2468] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea212a715184edab5c65be2f1e7b542 in term 0.
I20251026 09:11:12.341850 7958 leader_election.cc:304] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a8e2c3d243249e295b25cf18e0d8d58, 5ea212a715184edab5c65be2f1e7b542; no voters:
I20251026 09:11:12.341961 8122 raft_consensus.cc:2804] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:12.341997 8122 raft_consensus.cc:493] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:12.342028 8122 raft_consensus.cc:3060] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:12.342726 8122 raft_consensus.cc:515] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.342886 8122 leader_election.cc:290] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [CANDIDATE]: Term 1 election: Requested vote from peers 5a8e2c3d243249e295b25cf18e0d8d58 (127.3.14.1:38763), e6659534443548dca10d69a62a2e0fa1 (127.3.14.3:45531)
I20251026 09:11:12.343062 8067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90e2755d45eb496f89852bdb34bb421b" candidate_uuid: "5ea212a715184edab5c65be2f1e7b542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6659534443548dca10d69a62a2e0fa1"
I20251026 09:11:12.343102 7916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90e2755d45eb496f89852bdb34bb421b" candidate_uuid: "5ea212a715184edab5c65be2f1e7b542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8e2c3d243249e295b25cf18e0d8d58"
I20251026 09:11:12.343144 8067 raft_consensus.cc:3060] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:12.343148 7916 raft_consensus.cc:3060] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:12.343916 7916 raft_consensus.cc:2468] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea212a715184edab5c65be2f1e7b542 in term 1.
I20251026 09:11:12.343916 8067 raft_consensus.cc:2468] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea212a715184edab5c65be2f1e7b542 in term 1.
I20251026 09:11:12.344157 7958 leader_election.cc:304] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a8e2c3d243249e295b25cf18e0d8d58, 5ea212a715184edab5c65be2f1e7b542; no voters:
I20251026 09:11:12.344286 8122 raft_consensus.cc:2804] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:12.344342 8122 raft_consensus.cc:697] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Becoming Leader. State: Replica: 5ea212a715184edab5c65be2f1e7b542, State: Running, Role: LEADER
I20251026 09:11:12.344403 8122 consensus_queue.cc:237] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:12.345023 7813 catalog_manager.cc:5649] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "5ea212a715184edab5c65be2f1e7b542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:12.357787 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:12.359480 8138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:12.359571 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:12.359467 8140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:12.359467 8137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:12.359961 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:12.359997 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:12.360010 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469872360010 us; error 0 us; skew 500 ppm
I20251026 09:11:12.360782 3128 webserver.cc:492] Webserver started at http://127.3.14.4:35919/ using document root <none> and password file <none>
I20251026 09:11:12.360868 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:12.360906 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:12.360968 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:12.361275 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-3-root/instance:
uuid: "27d8ba8144a747d191e54cf85b67c4b9"
format_stamp: "Formatted at 2025-10-26 09:11:12 on dist-test-slave-w8k0"
I20251026 09:11:12.362263 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:12.362807 8145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.362931 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.362985 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-3-root
uuid: "27d8ba8144a747d191e54cf85b67c4b9"
format_stamp: "Formatted at 2025-10-26 09:11:12 on dist-test-slave-w8k0"
I20251026 09:11:12.363035 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:12.389438 8122 consensus_queue.cc:1048] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:12.390151 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:12.390432 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
W20251026 09:11:12.390581 8128 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20251026 09:11:12.390641 8128 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20251026 09:11:12.390692 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:12.390936 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:12.390975 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.391037 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:12.391068 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:12.394606 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.4:34279
I20251026 09:11:12.394649 8210 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.4:34279 every 8 connection(s)
I20251026 09:11:12.398587 8211 heartbeater.cc:344] Connected to a master server at 127.3.14.62:37769
I20251026 09:11:12.398674 8211 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:12.398778 8211 heartbeater.cc:507] Master 127.3.14.62:37769 requested a full tablet report, sending...
I20251026 09:11:12.399041 7813 ts_manager.cc:194] Registered new tserver with Master: 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4:34279)
I20251026 09:11:12.399493 7813 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54146
I20251026 09:11:12.404667 8122 consensus_queue.cc:1048] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:12.421846 8122 consensus_queue.cc:1048] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:12.422086 8118 consensus_queue.cc:1048] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:11:12.433490 7943 tablet.cc:2378] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 09:11:12.435549 8118 consensus_queue.cc:1048] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:12.467550 8128 consensus_queue.cc:1048] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:13.126001 7867 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:13.129619 7992 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e27b2e91f46f460a98311ae73d00c61c"
dest_uuid: "5ea212a715184edab5c65be2f1e7b542"
mode: GRACEFUL
new_leader_uuid: "5a8e2c3d243249e295b25cf18e0d8d58"
from {username='slave'} at 127.0.0.1:58654
I20251026 09:11:13.129753 7992 raft_consensus.cc:606] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Received request to transfer leadership to TS 5a8e2c3d243249e295b25cf18e0d8d58
I20251026 09:11:13.129943 7867 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
W20251026 09:11:13.129980 7860 auto_rebalancer.cc:253] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20251026 09:11:13.129992 7867 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 09:11:13.130477 7860 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e27b2e91f46f460a98311ae73d00c61c, TS 5a8e2c3d243249e295b25cf18e0d8d58 -> TS 27d8ba8144a747d191e54cf85b67c4b9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
W20251026 09:11:13.130515 7860 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet e27b2e91f46f460a98311ae73d00c61c, TS 5a8e2c3d243249e295b25cf18e0d8d58 -> TS 27d8ba8144a747d191e54cf85b67c4b9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
W20251026 09:11:13.130921 7860 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 37609a723dc649ccb93995d750aa7854, TS 5ea212a715184edab5c65be2f1e7b542 -> TS 27d8ba8144a747d191e54cf85b67c4b9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
W20251026 09:11:13.130960 7860 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 37609a723dc649ccb93995d750aa7854, TS 5ea212a715184edab5c65be2f1e7b542 -> TS 27d8ba8144a747d191e54cf85b67c4b9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:13.162062 8224 raft_consensus.cc:993] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542: : Instructing follower 5a8e2c3d243249e295b25cf18e0d8d58 to start an election
I20251026 09:11:13.162185 8224 raft_consensus.cc:1081] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Signalling peer 5a8e2c3d243249e295b25cf18e0d8d58 to start an election
I20251026 09:11:13.162448 7916 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e27b2e91f46f460a98311ae73d00c61c"
dest_uuid: "5a8e2c3d243249e295b25cf18e0d8d58"
from {username='slave'} at 127.0.0.1:42794
I20251026 09:11:13.162528 7916 raft_consensus.cc:493] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 09:11:13.162551 7916 raft_consensus.cc:3060] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:13.163237 7916 raft_consensus.cc:515] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:13.163478 7916 leader_election.cc:290] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [CANDIDATE]: Term 2 election: Requested vote from peers 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2:43439), e6659534443548dca10d69a62a2e0fa1 (127.3.14.3:45531)
I20251026 09:11:13.166328 8067 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e27b2e91f46f460a98311ae73d00c61c" candidate_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e6659534443548dca10d69a62a2e0fa1"
I20251026 09:11:13.166321 7991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e27b2e91f46f460a98311ae73d00c61c" candidate_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ea212a715184edab5c65be2f1e7b542"
I20251026 09:11:13.166471 8067 raft_consensus.cc:3060] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:13.166477 7991 raft_consensus.cc:3055] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 09:11:13.166528 7991 raft_consensus.cc:740] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ea212a715184edab5c65be2f1e7b542, State: Running, Role: LEADER
I20251026 09:11:13.166608 7991 consensus_queue.cc:260] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:13.166712 7991 raft_consensus.cc:3060] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:13.167430 8067 raft_consensus.cc:2468] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a8e2c3d243249e295b25cf18e0d8d58 in term 2.
I20251026 09:11:13.167483 7991 raft_consensus.cc:2468] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a8e2c3d243249e295b25cf18e0d8d58 in term 2.
I20251026 09:11:13.167657 7879 leader_election.cc:304] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a8e2c3d243249e295b25cf18e0d8d58, e6659534443548dca10d69a62a2e0fa1; no voters:
I20251026 09:11:13.167887 8233 raft_consensus.cc:2804] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:11:13.168040 8233 raft_consensus.cc:697] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 2 LEADER]: Becoming Leader. State: Replica: 5a8e2c3d243249e295b25cf18e0d8d58, State: Running, Role: LEADER
I20251026 09:11:13.168108 8233 consensus_queue.cc:237] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } }
I20251026 09:11:13.168650 7812 catalog_manager.cc:5649] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 reported cstate change: term changed from 1 to 2, leader changed from 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2) to 5a8e2c3d243249e295b25cf18e0d8d58 (127.3.14.1). New cstate: current_term: 2 leader_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:13.261211 8067 raft_consensus.cc:1275] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 2 FOLLOWER]: Refusing update from remote peer 5a8e2c3d243249e295b25cf18e0d8d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:11:13.261485 8233 consensus_queue.cc:1048] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:13.278007 7991 raft_consensus.cc:1275] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 2 FOLLOWER]: Refusing update from remote peer 5a8e2c3d243249e295b25cf18e0d8d58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:11:13.278373 8234 consensus_queue.cc:1048] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:13.400034 8211 heartbeater.cc:499] Master 127.3.14.62:37769 was elected leader, sending a full tablet report...
I20251026 09:11:14.132066 7991 consensus_queue.cc:237] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } }
I20251026 09:11:14.132826 8067 raft_consensus.cc:1275] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Refusing update from remote peer 5ea212a715184edab5c65be2f1e7b542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:11:14.132874 7916 raft_consensus.cc:1275] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Refusing update from remote peer 5ea212a715184edab5c65be2f1e7b542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:11:14.133059 8224 consensus_queue.cc:1048] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:14.133222 8244 consensus_queue.cc:1048] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:14.134171 7916 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } } }
I20251026 09:11:14.134292 8067 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } } }
I20251026 09:11:14.135552 7813 catalog_manager.cc:5649] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 reported cstate change: config changed from index -1 to 2, NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "5ea212a715184edab5c65be2f1e7b542" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } } }
W20251026 09:11:14.136483 7957 consensus_peers.cc:597] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 -> Peer 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4:34279): Couldn't send request to peer 27d8ba8144a747d191e54cf85b67c4b9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90e2755d45eb496f89852bdb34bb421b. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:14.187194 8244 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } } }
I20251026 09:11:14.191980 8067 consensus_queue.cc:237] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } }
I20251026 09:11:14.192756 7991 raft_consensus.cc:1275] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Refusing update from remote peer e6659534443548dca10d69a62a2e0fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:11:14.192760 7916 raft_consensus.cc:1275] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Refusing update from remote peer e6659534443548dca10d69a62a2e0fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:11:14.193117 8256 consensus_queue.cc:1048] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:14.193261 8255 consensus_queue.cc:1048] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:14.194129 7991 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } } }
I20251026 09:11:14.194649 7916 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } } }
W20251026 09:11:14.195948 8032 consensus_peers.cc:597] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 -> Peer 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4:34279): Couldn't send request to peer 27d8ba8144a747d191e54cf85b67c4b9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37609a723dc649ccb93995d750aa7854. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:14.196197 7812 catalog_manager.cc:5649] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 reported cstate change: config changed from index -1 to 2, NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "e6659534443548dca10d69a62a2e0fa1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } } }
I20251026 09:11:14.197175 8134 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } } }
I20251026 09:11:14.229111 8262 ts_tablet_manager.cc:933] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: Initiating tablet copy from peer 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2:43439)
I20251026 09:11:14.229379 8262 tablet_copy_client.cc:323] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:43439
I20251026 09:11:14.232987 8002 tablet_copy_service.cc:140] P 5ea212a715184edab5c65be2f1e7b542: Received BeginTabletCopySession request for tablet 90e2755d45eb496f89852bdb34bb421b from peer 27d8ba8144a747d191e54cf85b67c4b9 ({username='slave'} at 127.0.0.1:58672)
I20251026 09:11:14.233094 8002 tablet_copy_service.cc:161] P 5ea212a715184edab5c65be2f1e7b542: Beginning new tablet copy session on tablet 90e2755d45eb496f89852bdb34bb421b from peer 27d8ba8144a747d191e54cf85b67c4b9 at {username='slave'} at 127.0.0.1:58672: session id = 27d8ba8144a747d191e54cf85b67c4b9-90e2755d45eb496f89852bdb34bb421b
I20251026 09:11:14.233753 8002 tablet_copy_source_session.cc:215] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:11:14.234108 8262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90e2755d45eb496f89852bdb34bb421b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:14.235380 8262 tablet_copy_client.cc:806] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: tablet copy: Starting download of 0 data blocks...
I20251026 09:11:14.235559 8262 tablet_copy_client.cc:670] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: tablet copy: Starting download of 1 WAL segments...
I20251026 09:11:14.236145 8262 tablet_copy_client.cc:538] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:14.236932 8262 tablet_bootstrap.cc:492] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: Bootstrap starting.
I20251026 09:11:14.238446 8262 tablet_bootstrap.cc:492] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:14.238626 8262 tablet_bootstrap.cc:492] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: Bootstrap complete.
I20251026 09:11:14.238740 8262 ts_tablet_manager.cc:1403] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251026 09:11:14.238927 8262 raft_consensus.cc:359] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } }
I20251026 09:11:14.238997 8262 raft_consensus.cc:740] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 27d8ba8144a747d191e54cf85b67c4b9, State: Initialized, Role: LEARNER
I20251026 09:11:14.239024 8262 consensus_queue.cc:260] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } }
I20251026 09:11:14.239225 8262 ts_tablet_manager.cc:1434] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:14.239444 8002 tablet_copy_service.cc:342] P 5ea212a715184edab5c65be2f1e7b542: Request end of tablet copy session 27d8ba8144a747d191e54cf85b67c4b9-90e2755d45eb496f89852bdb34bb421b received from {username='slave'} at 127.0.0.1:58672
I20251026 09:11:14.239500 8002 tablet_copy_service.cc:434] P 5ea212a715184edab5c65be2f1e7b542: ending tablet copy session 27d8ba8144a747d191e54cf85b67c4b9-90e2755d45eb496f89852bdb34bb421b on tablet 90e2755d45eb496f89852bdb34bb421b with peer 27d8ba8144a747d191e54cf85b67c4b9
I20251026 09:11:14.314450 8262 ts_tablet_manager.cc:933] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: Initiating tablet copy from peer e6659534443548dca10d69a62a2e0fa1 (127.3.14.3:45531)
I20251026 09:11:14.314672 8262 tablet_copy_client.cc:323] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:45531
I20251026 09:11:14.317451 8077 tablet_copy_service.cc:140] P e6659534443548dca10d69a62a2e0fa1: Received BeginTabletCopySession request for tablet 37609a723dc649ccb93995d750aa7854 from peer 27d8ba8144a747d191e54cf85b67c4b9 ({username='slave'} at 127.0.0.1:53964)
I20251026 09:11:14.317533 8077 tablet_copy_service.cc:161] P e6659534443548dca10d69a62a2e0fa1: Beginning new tablet copy session on tablet 37609a723dc649ccb93995d750aa7854 from peer 27d8ba8144a747d191e54cf85b67c4b9 at {username='slave'} at 127.0.0.1:53964: session id = 27d8ba8144a747d191e54cf85b67c4b9-37609a723dc649ccb93995d750aa7854
I20251026 09:11:14.318154 8077 tablet_copy_source_session.cc:215] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:11:14.318534 8262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37609a723dc649ccb93995d750aa7854. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:14.320185 8262 tablet_copy_client.cc:806] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: tablet copy: Starting download of 0 data blocks...
I20251026 09:11:14.320395 8262 tablet_copy_client.cc:670] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: tablet copy: Starting download of 1 WAL segments...
I20251026 09:11:14.320950 8262 tablet_copy_client.cc:538] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:14.321975 8262 tablet_bootstrap.cc:492] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: Bootstrap starting.
I20251026 09:11:14.323544 8262 tablet_bootstrap.cc:492] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:14.323746 8262 tablet_bootstrap.cc:492] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: Bootstrap complete.
I20251026 09:11:14.323864 8262 ts_tablet_manager.cc:1403] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251026 09:11:14.324040 8262 raft_consensus.cc:359] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } }
I20251026 09:11:14.324110 8262 raft_consensus.cc:740] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 27d8ba8144a747d191e54cf85b67c4b9, State: Initialized, Role: LEARNER
I20251026 09:11:14.324144 8262 consensus_queue.cc:260] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: true } }
I20251026 09:11:14.324298 8262 ts_tablet_manager.cc:1434] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:14.324491 8077 tablet_copy_service.cc:342] P e6659534443548dca10d69a62a2e0fa1: Request end of tablet copy session 27d8ba8144a747d191e54cf85b67c4b9-37609a723dc649ccb93995d750aa7854 received from {username='slave'} at 127.0.0.1:53964
I20251026 09:11:14.324540 8077 tablet_copy_service.cc:434] P e6659534443548dca10d69a62a2e0fa1: ending tablet copy session 27d8ba8144a747d191e54cf85b67c4b9-37609a723dc649ccb93995d750aa7854 on tablet 37609a723dc649ccb93995d750aa7854 with peer 27d8ba8144a747d191e54cf85b67c4b9
I20251026 09:11:14.346633 8185 raft_consensus.cc:1217] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 09:11:14.424119 8185 raft_consensus.cc:1217] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 09:11:14.461112 8251 raft_consensus.cc:1064] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542: attempting to promote NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 to VOTER
I20251026 09:11:14.461432 8251 consensus_queue.cc:237] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } }
I20251026 09:11:14.462009 8066 raft_consensus.cc:1275] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Refusing update from remote peer 5ea212a715184edab5c65be2f1e7b542: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:14.462066 8185 raft_consensus.cc:1275] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 LEARNER]: Refusing update from remote peer 5ea212a715184edab5c65be2f1e7b542: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:14.462222 7916 raft_consensus.cc:1275] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Refusing update from remote peer 5ea212a715184edab5c65be2f1e7b542: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:14.462282 8224 consensus_queue.cc:1048] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:14.462417 8243 consensus_queue.cc:1048] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:14.462626 8224 consensus_queue.cc:1048] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:14.463784 7916 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.464701 8185 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.465950 7813 catalog_manager.cc:5649] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 reported cstate change: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ea212a715184edab5c65be2f1e7b542" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.487540 8251 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.489042 8066 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } attrs { replace: true } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.490717 7991 consensus_queue.cc:237] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } }
I20251026 09:11:14.491199 7916 raft_consensus.cc:1275] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Refusing update from remote peer 5ea212a715184edab5c65be2f1e7b542: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:14.491225 8185 raft_consensus.cc:1275] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Refusing update from remote peer 5ea212a715184edab5c65be2f1e7b542: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:14.491488 8271 consensus_queue.cc:1048] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:14.491676 8244 consensus_queue.cc:1048] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:14.492612 7916 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e6659534443548dca10d69a62a2e0fa1 (127.3.14.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.492619 8185 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e6659534443548dca10d69a62a2e0fa1 (127.3.14.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.493872 7811 catalog_manager.cc:5649] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 reported cstate change: config changed from index 3 to 4, VOTER e6659534443548dca10d69a62a2e0fa1 (127.3.14.3) evicted. New cstate: current_term: 1 leader_uuid: "5ea212a715184edab5c65be2f1e7b542" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.529052 8057 tablet_service.cc:1552] Processing DeleteTablet for tablet 90e2755d45eb496f89852bdb34bb421b with delete_type TABLET_DATA_TOMBSTONED (TS e6659534443548dca10d69a62a2e0fa1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53938
I20251026 09:11:14.529390 8272 tablet_replica.cc:333] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1: stopping tablet replica
I20251026 09:11:14.529487 8272 raft_consensus.cc:2243] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:14.529594 8272 raft_consensus.cc:2272] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:14.529924 8272 ts_tablet_manager.cc:1916] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:11:14.532234 8272 ts_tablet_manager.cc:1929] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 09:11:14.532328 8272 log.cc:1199] T 90e2755d45eb496f89852bdb34bb421b P e6659534443548dca10d69a62a2e0fa1: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/90e2755d45eb496f89852bdb34bb421b
I20251026 09:11:14.532768 7797 catalog_manager.cc:4985] TS e6659534443548dca10d69a62a2e0fa1 (127.3.14.3:45531): tablet 90e2755d45eb496f89852bdb34bb421b (table test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]) successfully deleted
I20251026 09:11:14.626085 8134 raft_consensus.cc:1064] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1: attempting to promote NON_VOTER 27d8ba8144a747d191e54cf85b67c4b9 to VOTER
I20251026 09:11:14.626339 8134 consensus_queue.cc:237] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } }
I20251026 09:11:14.626765 8185 raft_consensus.cc:1275] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 LEARNER]: Refusing update from remote peer e6659534443548dca10d69a62a2e0fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:14.626809 7916 raft_consensus.cc:1275] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Refusing update from remote peer e6659534443548dca10d69a62a2e0fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:14.626765 7992 raft_consensus.cc:1275] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Refusing update from remote peer e6659534443548dca10d69a62a2e0fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:14.626997 8255 consensus_queue.cc:1048] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:14.627121 8261 consensus_queue.cc:1048] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:14.627245 8255 consensus_queue.cc:1048] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:14.627928 7991 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.628259 8185 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.629361 7811 catalog_manager.cc:5649] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 reported cstate change: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6659534443548dca10d69a62a2e0fa1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.676141 8224 raft_consensus.cc:2955] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e6659534443548dca10d69a62a2e0fa1 (127.3.14.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.677440 7800 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 90e2755d45eb496f89852bdb34bb421b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:11:14.684924 8255 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.686420 7916 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 27d8ba8144a747d191e54cf85b67c4b9 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "5ea212a715184edab5c65be2f1e7b542" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 43439 } attrs { replace: true } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.702790 8066 consensus_queue.cc:237] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } }
I20251026 09:11:14.703285 8185 raft_consensus.cc:1275] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Refusing update from remote peer e6659534443548dca10d69a62a2e0fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:14.703392 7916 raft_consensus.cc:1275] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Refusing update from remote peer e6659534443548dca10d69a62a2e0fa1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:14.703553 8255 consensus_queue.cc:1048] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:14.703696 8255 consensus_queue.cc:1048] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:14.704384 8185 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.704421 7916 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.705838 7813 catalog_manager.cc:5649] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 reported cstate change: config changed from index 3 to 4, VOTER 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2) evicted. New cstate: current_term: 1 leader_uuid: "e6659534443548dca10d69a62a2e0fa1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.797817 7980 tablet_service.cc:1552] Processing DeleteTablet for tablet 37609a723dc649ccb93995d750aa7854 with delete_type TABLET_DATA_TOMBSTONED (TS 5ea212a715184edab5c65be2f1e7b542 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58632
I20251026 09:11:14.798116 8275 tablet_replica.cc:333] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542: stopping tablet replica
I20251026 09:11:14.798194 8275 raft_consensus.cc:2243] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:14.798287 8275 raft_consensus.cc:2272] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:14.798573 8275 ts_tablet_manager.cc:1916] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:11:14.800625 8275 ts_tablet_manager.cc:1929] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 09:11:14.800767 8275 log.cc:1199] T 37609a723dc649ccb93995d750aa7854 P 5ea212a715184edab5c65be2f1e7b542: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/37609a723dc649ccb93995d750aa7854
I20251026 09:11:14.801196 7800 catalog_manager.cc:4985] TS 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2:43439): tablet 37609a723dc649ccb93995d750aa7854 (table test-workload [id=665e4c6e3dd8467d9f61082cf98b9586]) successfully deleted
I20251026 09:11:14.906531 8255 raft_consensus.cc:2955] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ea212a715184edab5c65be2f1e7b542 (127.3.14.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a8e2c3d243249e295b25cf18e0d8d58" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 38763 } } peers { permanent_uuid: "e6659534443548dca10d69a62a2e0fa1" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 45531 } } peers { permanent_uuid: "27d8ba8144a747d191e54cf85b67c4b9" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 34279 } attrs { promote: false } } }
I20251026 09:11:14.907634 7797 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 37609a723dc649ccb93995d750aa7854 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:11:15.130316 7867 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:15.130481 7867 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:15.130527 7867 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:17.130718 7867 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:17.130888 7867 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:17.130937 7867 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:19.131134 7867 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:19.131289 7867 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:19.131315 7867 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:21.131532 7867 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:21.131700 7867 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:21.131748 7867 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:21.471678 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:11:21.474745 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:21.474882 3128 tablet_replica.cc:333] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58: stopping tablet replica
I20251026 09:11:21.474978 3128 raft_consensus.cc:2243] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 2 LEADER]: Raft consensus shutting down.
I20251026 09:11:21.475056 3128 raft_consensus.cc:2272] T e27b2e91f46f460a98311ae73d00c61c P 5a8e2c3d243249e295b25cf18e0d8d58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:21.475234 3128 tablet_replica.cc:333] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58: stopping tablet replica
I20251026 09:11:21.475266 3128 raft_consensus.cc:2243] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:21.475308 3128 raft_consensus.cc:2272] T 37609a723dc649ccb93995d750aa7854 P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:21.475447 3128 tablet_replica.cc:333] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58: stopping tablet replica
I20251026 09:11:21.475482 3128 raft_consensus.cc:2243] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:21.475536 3128 raft_consensus.cc:2272] T 90e2755d45eb496f89852bdb34bb421b P 5a8e2c3d243249e295b25cf18e0d8d58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 09:11:21.485275 7958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.14.1:38763: connect: Connection refused (error 111)
W20251026 09:11:21.485743 7958 consensus_peers.cc:597] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 -> Peer 5a8e2c3d243249e295b25cf18e0d8d58 (127.3.14.1:38763): Couldn't send request to peer 5a8e2c3d243249e295b25cf18e0d8d58. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:38763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:21.487778 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:11:21.489078 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:11:21.492867 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:21.493191 3128 tablet_replica.cc:333] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542: stopping tablet replica
I20251026 09:11:21.493259 3128 raft_consensus.cc:2243] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:21.493297 3128 raft_consensus.cc:2272] T e27b2e91f46f460a98311ae73d00c61c P 5ea212a715184edab5c65be2f1e7b542 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:21.493491 3128 tablet_replica.cc:333] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542: stopping tablet replica
I20251026 09:11:21.493593 3128 raft_consensus.cc:2243] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:21.493664 3128 raft_consensus.cc:2272] T 90e2755d45eb496f89852bdb34bb421b P 5ea212a715184edab5c65be2f1e7b542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:21.506007 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:11:21.507436 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:11:21.511538 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:21.511724 3128 tablet_replica.cc:333] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1: stopping tablet replica
I20251026 09:11:21.511808 3128 raft_consensus.cc:2243] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:21.511859 3128 raft_consensus.cc:2272] T e27b2e91f46f460a98311ae73d00c61c P e6659534443548dca10d69a62a2e0fa1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:21.512060 3128 tablet_replica.cc:333] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1: stopping tablet replica
I20251026 09:11:21.512101 3128 raft_consensus.cc:2243] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:21.512158 3128 raft_consensus.cc:2272] T 37609a723dc649ccb93995d750aa7854 P e6659534443548dca10d69a62a2e0fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:21.524363 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:11:21.525588 3128 tablet_server.cc:178] TabletServer@127.3.14.4:0 shutting down...
I20251026 09:11:21.528851 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:21.528985 3128 tablet_replica.cc:333] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9: stopping tablet replica
I20251026 09:11:21.529057 3128 raft_consensus.cc:2243] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:21.529121 3128 raft_consensus.cc:2272] T 37609a723dc649ccb93995d750aa7854 P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:21.529332 3128 tablet_replica.cc:333] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9: stopping tablet replica
I20251026 09:11:21.529378 3128 raft_consensus.cc:2243] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:21.529426 3128 raft_consensus.cc:2272] T 90e2755d45eb496f89852bdb34bb421b P 27d8ba8144a747d191e54cf85b67c4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:21.542028 3128 tablet_server.cc:195] TabletServer@127.3.14.4:0 shutdown complete.
I20251026 09:11:21.543301 3128 master.cc:561] Master@127.3.14.62:37769 shutting down...
I20251026 09:11:21.546468 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:21.546574 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:21.546595 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 65745862afe44f4d9f82934393ef6f60: stopping tablet replica
I20251026 09:11:21.559141 3128 master.cc:583] Master@127.3.14.62:37769 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10479 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20251026 09:11:21.565742 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:43505
I20251026 09:11:21.565907 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:21.567557 8282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:21.567557 8285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:21.567663 8283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:21.567695 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:21.568049 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:21.568092 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:21.568125 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469881568124 us; error 0 us; skew 500 ppm
I20251026 09:11:21.568828 3128 webserver.cc:492] Webserver started at http://127.3.14.62:40555/ using document root <none> and password file <none>
I20251026 09:11:21.568926 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:21.568979 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:21.569046 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:21.569372 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "a64efaa7d0c647da92737c904b3084a7"
format_stamp: "Formatted at 2025-10-26 09:11:21 on dist-test-slave-w8k0"
I20251026 09:11:21.570407 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:21.571022 8290 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.571224 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:21.571273 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "a64efaa7d0c647da92737c904b3084a7"
format_stamp: "Formatted at 2025-10-26 09:11:21 on dist-test-slave-w8k0"
I20251026 09:11:21.571494 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:21.583081 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:21.583436 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:21.586153 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:43505
I20251026 09:11:21.589710 8342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:43505 every 8 connection(s)
I20251026 09:11:21.589946 8343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:21.591085 8343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7: Bootstrap starting.
I20251026 09:11:21.591475 8343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:21.592068 8343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7: No bootstrap required, opened a new log
I20251026 09:11:21.592211 8343 raft_consensus.cc:359] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64efaa7d0c647da92737c904b3084a7" member_type: VOTER }
I20251026 09:11:21.592269 8343 raft_consensus.cc:385] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:21.592293 8343 raft_consensus.cc:740] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64efaa7d0c647da92737c904b3084a7, State: Initialized, Role: FOLLOWER
I20251026 09:11:21.592350 8343 consensus_queue.cc:260] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64efaa7d0c647da92737c904b3084a7" member_type: VOTER }
I20251026 09:11:21.592396 8343 raft_consensus.cc:399] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:11:21.592423 8343 raft_consensus.cc:493] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:11:21.592456 8343 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:21.592995 8343 raft_consensus.cc:515] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64efaa7d0c647da92737c904b3084a7" member_type: VOTER }
I20251026 09:11:21.593060 8343 leader_election.cc:304] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a64efaa7d0c647da92737c904b3084a7; no voters:
I20251026 09:11:21.593202 8343 leader_election.cc:290] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:11:21.593276 8346 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:21.593382 8346 raft_consensus.cc:697] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 1 LEADER]: Becoming Leader. State: Replica: a64efaa7d0c647da92737c904b3084a7, State: Running, Role: LEADER
I20251026 09:11:21.593387 8343 sys_catalog.cc:565] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:11:21.593487 8346 consensus_queue.cc:237] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64efaa7d0c647da92737c904b3084a7" member_type: VOTER }
I20251026 09:11:21.593771 8348 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a64efaa7d0c647da92737c904b3084a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64efaa7d0c647da92737c904b3084a7" member_type: VOTER } }
I20251026 09:11:21.593806 8347 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a64efaa7d0c647da92737c904b3084a7. Latest consensus state: current_term: 1 leader_uuid: "a64efaa7d0c647da92737c904b3084a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64efaa7d0c647da92737c904b3084a7" member_type: VOTER } }
I20251026 09:11:21.594002 8347 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:21.593900 8348 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:21.594287 8354 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:11:21.594439 8354 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:11:21.595002 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:11:21.595014 8354 catalog_manager.cc:1357] Generated new cluster ID: f2e3e4660f244f5e88b69a7029db77f4
I20251026 09:11:21.595073 8354 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:11:21.607954 8354 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:11:21.608405 8354 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:11:21.616029 8354 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7: Generated new TSK 0
I20251026 09:11:21.616165 8354 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:11:21.627013 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:11:21.628702 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:21.628794 8367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:21.628814 8365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:21.628819 8364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:21.629104 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:21.629146 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:21.629168 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469881629167 us; error 0 us; skew 500 ppm
I20251026 09:11:21.629796 3128 webserver.cc:492] Webserver started at http://127.3.14.1:45753/ using document root <none> and password file <none>
I20251026 09:11:21.629915 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:21.629971 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:21.630023 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:21.630331 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "c054951c209c4dc2913f912c889f0255"
format_stamp: "Formatted at 2025-10-26 09:11:21 on dist-test-slave-w8k0"
I20251026 09:11:21.631381 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:21.631909 8372 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.632043 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:21.632089 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "c054951c209c4dc2913f912c889f0255"
format_stamp: "Formatted at 2025-10-26 09:11:21 on dist-test-slave-w8k0"
I20251026 09:11:21.632149 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:21.639398 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:21.639698 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:21.639868 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:21.640086 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:21.640116 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.640153 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:21.640184 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.642969 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:33973
I20251026 09:11:21.643031 8435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:33973 every 8 connection(s)
I20251026 09:11:21.643425 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:21.644752 8443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:21.644812 8441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:21.644891 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:21.644783 8440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:21.645215 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:21.645254 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:21.645272 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469881645272 us; error 0 us; skew 500 ppm
I20251026 09:11:21.645936 3128 webserver.cc:492] Webserver started at http://127.3.14.2:33067/ using document root <none> and password file <none>
I20251026 09:11:21.646027 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:21.646080 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:21.646152 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:21.646514 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "20fc63609b7747c6ac4b0f2d7500b996"
format_stamp: "Formatted at 2025-10-26 09:11:21 on dist-test-slave-w8k0"
I20251026 09:11:21.647795 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 09:11:21.648384 8448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.648581 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:21.648636 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "20fc63609b7747c6ac4b0f2d7500b996"
format_stamp: "Formatted at 2025-10-26 09:11:21 on dist-test-slave-w8k0"
I20251026 09:11:21.648700 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:21.650624 8436 heartbeater.cc:344] Connected to a master server at 127.3.14.62:43505
I20251026 09:11:21.650693 8436 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:21.650816 8436 heartbeater.cc:507] Master 127.3.14.62:43505 requested a full tablet report, sending...
I20251026 09:11:21.651100 8307 ts_manager.cc:194] Registered new tserver with Master: c054951c209c4dc2913f912c889f0255 (127.3.14.1:33973)
I20251026 09:11:21.651683 8307 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60414
I20251026 09:11:21.665802 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:21.666155 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:21.666347 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:21.666566 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:21.666596 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.666635 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:21.666667 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.669535 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:38669
I20251026 09:11:21.669589 8511 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:38669 every 8 connection(s)
I20251026 09:11:21.669971 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:21.671301 8515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:21.671350 8516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:21.671435 8518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:21.671587 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:21.671739 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:21.671775 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:21.671793 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469881671794 us; error 0 us; skew 500 ppm
I20251026 09:11:21.672566 3128 webserver.cc:492] Webserver started at http://127.3.14.3:45983/ using document root <none> and password file <none>
I20251026 09:11:21.672669 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:21.672719 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:21.672773 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:21.673178 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "c8675077caef476b9d387da8c094f867"
format_stamp: "Formatted at 2025-10-26 09:11:21 on dist-test-slave-w8k0"
I20251026 09:11:21.673450 8512 heartbeater.cc:344] Connected to a master server at 127.3.14.62:43505
I20251026 09:11:21.673509 8512 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:21.673620 8512 heartbeater.cc:507] Master 127.3.14.62:43505 requested a full tablet report, sending...
I20251026 09:11:21.673846 8307 ts_manager.cc:194] Registered new tserver with Master: 20fc63609b7747c6ac4b0f2d7500b996 (127.3.14.2:38669)
I20251026 09:11:21.674218 8307 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60424
I20251026 09:11:21.674382 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:21.674960 8523 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.675096 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:21.675139 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "c8675077caef476b9d387da8c094f867"
format_stamp: "Formatted at 2025-10-26 09:11:21 on dist-test-slave-w8k0"
I20251026 09:11:21.675194 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:21.697937 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:21.698180 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:21.698397 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:21.698635 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:21.698665 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.698707 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:21.698738 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:21.702067 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:41389
I20251026 09:11:21.702162 8586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:41389 every 8 connection(s)
I20251026 09:11:21.705582 8587 heartbeater.cc:344] Connected to a master server at 127.3.14.62:43505
I20251026 09:11:21.705683 8587 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:21.705806 8587 heartbeater.cc:507] Master 127.3.14.62:43505 requested a full tablet report, sending...
I20251026 09:11:21.706214 8307 ts_manager.cc:194] Registered new tserver with Master: c8675077caef476b9d387da8c094f867 (127.3.14.3:41389)
I20251026 09:11:21.706599 8307 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60432
I20251026 09:11:21.706730 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004445966s
I20251026 09:11:22.652220 8436 heartbeater.cc:499] Master 127.3.14.62:43505 was elected leader, sending a full tablet report...
I20251026 09:11:22.674830 8512 heartbeater.cc:499] Master 127.3.14.62:43505 was elected leader, sending a full tablet report...
I20251026 09:11:22.707275 8587 heartbeater.cc:499] Master 127.3.14.62:43505 was elected leader, sending a full tablet report...
I20251026 09:11:22.732352 3128 test_util.cc:276] Using random seed: 715427645
I20251026 09:11:22.738269 8307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60440:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:11:22.738431 8307 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:11:22.744113 8551 tablet_service.cc:1505] Processing CreateTablet for tablet b5871821fd0946cdb79851526fd72df6 (DEFAULT_TABLE table=test-workload [id=ffa66ac090a749f1af644cac729bbe74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 09:11:22.744280 8551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5871821fd0946cdb79851526fd72df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:22.744997 8400 tablet_service.cc:1505] Processing CreateTablet for tablet b5871821fd0946cdb79851526fd72df6 (DEFAULT_TABLE table=test-workload [id=ffa66ac090a749f1af644cac729bbe74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 09:11:22.745114 8400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5871821fd0946cdb79851526fd72df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:22.745363 8476 tablet_service.cc:1505] Processing CreateTablet for tablet b5871821fd0946cdb79851526fd72df6 (DEFAULT_TABLE table=test-workload [id=ffa66ac090a749f1af644cac729bbe74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 09:11:22.745483 8476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5871821fd0946cdb79851526fd72df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:22.745962 8607 tablet_bootstrap.cc:492] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867: Bootstrap starting.
I20251026 09:11:22.746497 8607 tablet_bootstrap.cc:654] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:22.746543 8608 tablet_bootstrap.cc:492] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255: Bootstrap starting.
I20251026 09:11:22.747038 8608 tablet_bootstrap.cc:654] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:22.747038 8610 tablet_bootstrap.cc:492] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996: Bootstrap starting.
I20251026 09:11:22.747400 8607 tablet_bootstrap.cc:492] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867: No bootstrap required, opened a new log
I20251026 09:11:22.747479 8607 ts_tablet_manager.cc:1403] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:22.747668 8607 raft_consensus.cc:359] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } }
I20251026 09:11:22.747768 8607 raft_consensus.cc:385] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:22.747773 8610 tablet_bootstrap.cc:654] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:22.747824 8607 raft_consensus.cc:740] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8675077caef476b9d387da8c094f867, State: Initialized, Role: FOLLOWER
I20251026 09:11:22.747915 8607 consensus_queue.cc:260] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } }
I20251026 09:11:22.748144 8607 ts_tablet_manager.cc:1434] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:22.748245 8608 tablet_bootstrap.cc:492] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255: No bootstrap required, opened a new log
I20251026 09:11:22.748342 8608 ts_tablet_manager.cc:1403] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:11:22.748531 8608 raft_consensus.cc:359] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } }
I20251026 09:11:22.748623 8608 raft_consensus.cc:385] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:22.748672 8608 raft_consensus.cc:740] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c054951c209c4dc2913f912c889f0255, State: Initialized, Role: FOLLOWER
I20251026 09:11:22.748767 8608 consensus_queue.cc:260] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } }
I20251026 09:11:22.748764 8610 tablet_bootstrap.cc:492] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996: No bootstrap required, opened a new log
I20251026 09:11:22.748948 8610 ts_tablet_manager.cc:1403] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:22.749126 8610 raft_consensus.cc:359] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } }
I20251026 09:11:22.749212 8610 raft_consensus.cc:385] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:22.749238 8610 raft_consensus.cc:740] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20fc63609b7747c6ac4b0f2d7500b996, State: Initialized, Role: FOLLOWER
I20251026 09:11:22.749294 8610 consensus_queue.cc:260] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } }
I20251026 09:11:22.749456 8610 ts_tablet_manager.cc:1434] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:22.749109 8608 ts_tablet_manager.cc:1434] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:22.797612 8615 raft_consensus.cc:493] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:22.797747 8615 raft_consensus.cc:515] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } }
I20251026 09:11:22.798020 8615 leader_election.cc:290] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c054951c209c4dc2913f912c889f0255 (127.3.14.1:33973), c8675077caef476b9d387da8c094f867 (127.3.14.3:41389)
I20251026 09:11:22.801004 8561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5871821fd0946cdb79851526fd72df6" candidate_uuid: "20fc63609b7747c6ac4b0f2d7500b996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8675077caef476b9d387da8c094f867" is_pre_election: true
I20251026 09:11:22.800987 8410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5871821fd0946cdb79851526fd72df6" candidate_uuid: "20fc63609b7747c6ac4b0f2d7500b996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c054951c209c4dc2913f912c889f0255" is_pre_election: true
I20251026 09:11:22.801152 8561 raft_consensus.cc:2468] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20fc63609b7747c6ac4b0f2d7500b996 in term 0.
I20251026 09:11:22.801167 8410 raft_consensus.cc:2468] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20fc63609b7747c6ac4b0f2d7500b996 in term 0.
I20251026 09:11:22.801342 8449 leader_election.cc:304] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20fc63609b7747c6ac4b0f2d7500b996, c8675077caef476b9d387da8c094f867; no voters:
I20251026 09:11:22.801451 8615 raft_consensus.cc:2804] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:22.801519 8615 raft_consensus.cc:493] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:22.801538 8615 raft_consensus.cc:3060] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:22.802280 8615 raft_consensus.cc:515] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } }
I20251026 09:11:22.802433 8615 leader_election.cc:290] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [CANDIDATE]: Term 1 election: Requested vote from peers c054951c209c4dc2913f912c889f0255 (127.3.14.1:33973), c8675077caef476b9d387da8c094f867 (127.3.14.3:41389)
I20251026 09:11:22.802592 8410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5871821fd0946cdb79851526fd72df6" candidate_uuid: "20fc63609b7747c6ac4b0f2d7500b996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c054951c209c4dc2913f912c889f0255"
I20251026 09:11:22.802592 8561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5871821fd0946cdb79851526fd72df6" candidate_uuid: "20fc63609b7747c6ac4b0f2d7500b996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8675077caef476b9d387da8c094f867"
I20251026 09:11:22.802685 8410 raft_consensus.cc:3060] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:22.802707 8561 raft_consensus.cc:3060] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:22.803447 8561 raft_consensus.cc:2468] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20fc63609b7747c6ac4b0f2d7500b996 in term 1.
I20251026 09:11:22.803454 8410 raft_consensus.cc:2468] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20fc63609b7747c6ac4b0f2d7500b996 in term 1.
I20251026 09:11:22.803644 8451 leader_election.cc:304] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20fc63609b7747c6ac4b0f2d7500b996, c054951c209c4dc2913f912c889f0255; no voters:
I20251026 09:11:22.803789 8615 raft_consensus.cc:2804] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:22.804040 8615 raft_consensus.cc:697] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 1 LEADER]: Becoming Leader. State: Replica: 20fc63609b7747c6ac4b0f2d7500b996, State: Running, Role: LEADER
I20251026 09:11:22.804107 8615 consensus_queue.cc:237] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } }
I20251026 09:11:22.804699 8307 catalog_manager.cc:5649] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20fc63609b7747c6ac4b0f2d7500b996 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "20fc63609b7747c6ac4b0f2d7500b996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054951c209c4dc2913f912c889f0255" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 33973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20fc63609b7747c6ac4b0f2d7500b996" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 38669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:22.815567 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:11:22.818162 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:22.818395 3128 tablet_replica.cc:333] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255: stopping tablet replica
I20251026 09:11:22.818466 3128 raft_consensus.cc:2243] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:22.818516 3128 raft_consensus.cc:2272] T b5871821fd0946cdb79851526fd72df6 P c054951c209c4dc2913f912c889f0255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:22.830513 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:11:23.249090 8615 consensus_queue.cc:1048] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8675077caef476b9d387da8c094f867" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:11:23.251201 8451 consensus_peers.cc:597] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 -> Peer c054951c209c4dc2913f912c889f0255 (127.3.14.1:33973): Couldn't send request to peer c054951c209c4dc2913f912c889f0255. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:33973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:23.594978 8361 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:23.595108 8361 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:23.595139 8361 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:25.595355 8361 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:25.595489 8361 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:25.595510 8361 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 09:11:25.950184 8451 consensus_peers.cc:597] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 -> Peer c054951c209c4dc2913f912c889f0255 (127.3.14.1:33973): Couldn't send request to peer c054951c209c4dc2913f912c889f0255. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:33973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20251026 09:11:26.497088 8451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.14.1:33973: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20251026 09:11:27.595702 8361 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:27.595856 8361 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:27.595878 8361 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 09:11:28.611475 8451 consensus_peers.cc:597] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 -> Peer c054951c209c4dc2913f912c889f0255 (127.3.14.1:33973): Couldn't send request to peer c054951c209c4dc2913f912c889f0255. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:33973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251026 09:11:29.596082 8361 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:29.596230 8361 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:29.596269 8361 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 09:11:31.041163 8451 consensus_peers.cc:597] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 -> Peer c054951c209c4dc2913f912c889f0255 (127.3.14.1:33973): Couldn't send request to peer c054951c209c4dc2913f912c889f0255. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:33973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20251026 09:11:31.567252 8451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.14.1:33973: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251026 09:11:31.596444 8361 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:31.596592 8361 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:31.596614 8361 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 09:11:33.596809 8361 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 09:11:33.596956 8361 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 09:11:33.596982 8361 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 09:11:33.661999 8451 consensus_peers.cc:597] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 -> Peer c054951c209c4dc2913f912c889f0255 (127.3.14.1:33973): Couldn't send request to peer c054951c209c4dc2913f912c889f0255. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:33973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20251026 09:11:33.884265 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:11:33.887246 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:33.887416 3128 tablet_replica.cc:333] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996: stopping tablet replica
I20251026 09:11:33.887485 3128 raft_consensus.cc:2243] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:33.887579 3128 raft_consensus.cc:2272] T b5871821fd0946cdb79851526fd72df6 P 20fc63609b7747c6ac4b0f2d7500b996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:33.899763 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:11:33.900738 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:11:33.904294 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:33.904436 3128 tablet_replica.cc:333] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867: stopping tablet replica
I20251026 09:11:33.904506 3128 raft_consensus.cc:2243] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:33.904568 3128 raft_consensus.cc:2272] T b5871821fd0946cdb79851526fd72df6 P c8675077caef476b9d387da8c094f867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:33.916728 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:11:33.917737 3128 master.cc:561] Master@127.3.14.62:43505 shutting down...
I20251026 09:11:33.920358 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:33.920454 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:33.920475 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P a64efaa7d0c647da92737c904b3084a7: stopping tablet replica
I20251026 09:11:33.932904 3128 master.cc:583] Master@127.3.14.62:43505 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (12371 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20251026 09:11:33.937683 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:41575
I20251026 09:11:33.937851 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:33.939517 8655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:33.939512 8656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:33.939707 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:33.939718 8658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:33.939956 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:33.939996 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:33.940011 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469893940011 us; error 0 us; skew 500 ppm
I20251026 09:11:33.940651 3128 webserver.cc:492] Webserver started at http://127.3.14.62:44693/ using document root <none> and password file <none>
I20251026 09:11:33.940739 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:33.940820 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:33.940871 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:33.941170 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "ddedba7a0c7140b685e8f0ffdc6ed129"
format_stamp: "Formatted at 2025-10-26 09:11:33 on dist-test-slave-w8k0"
I20251026 09:11:33.942147 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:33.942716 8663 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:33.942893 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:33.942950 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "ddedba7a0c7140b685e8f0ffdc6ed129"
format_stamp: "Formatted at 2025-10-26 09:11:33 on dist-test-slave-w8k0"
I20251026 09:11:33.942999 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:33.948774 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:33.948994 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:33.951860 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:41575
I20251026 09:11:33.954159 8715 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:41575 every 8 connection(s)
I20251026 09:11:33.954337 8716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:33.955690 8716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129: Bootstrap starting.
I20251026 09:11:33.956056 8716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:33.956642 8716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129: No bootstrap required, opened a new log
I20251026 09:11:33.956785 8716 raft_consensus.cc:359] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddedba7a0c7140b685e8f0ffdc6ed129" member_type: VOTER }
I20251026 09:11:33.956849 8716 raft_consensus.cc:385] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:33.956874 8716 raft_consensus.cc:740] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddedba7a0c7140b685e8f0ffdc6ed129, State: Initialized, Role: FOLLOWER
I20251026 09:11:33.956945 8716 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddedba7a0c7140b685e8f0ffdc6ed129" member_type: VOTER }
I20251026 09:11:33.957001 8716 raft_consensus.cc:399] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:11:33.957031 8716 raft_consensus.cc:493] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:11:33.957063 8716 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:33.957682 8716 raft_consensus.cc:515] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddedba7a0c7140b685e8f0ffdc6ed129" member_type: VOTER }
I20251026 09:11:33.957767 8716 leader_election.cc:304] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddedba7a0c7140b685e8f0ffdc6ed129; no voters:
I20251026 09:11:33.957901 8716 leader_election.cc:290] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:11:33.957957 8719 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:33.958069 8716 sys_catalog.cc:565] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:11:33.958149 8719 raft_consensus.cc:697] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 1 LEADER]: Becoming Leader. State: Replica: ddedba7a0c7140b685e8f0ffdc6ed129, State: Running, Role: LEADER
I20251026 09:11:33.958217 8719 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddedba7a0c7140b685e8f0ffdc6ed129" member_type: VOTER }
I20251026 09:11:33.958482 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddedba7a0c7140b685e8f0ffdc6ed129. Latest consensus state: current_term: 1 leader_uuid: "ddedba7a0c7140b685e8f0ffdc6ed129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddedba7a0c7140b685e8f0ffdc6ed129" member_type: VOTER } }
I20251026 09:11:33.958518 8720 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddedba7a0c7140b685e8f0ffdc6ed129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddedba7a0c7140b685e8f0ffdc6ed129" member_type: VOTER } }
I20251026 09:11:33.958631 8720 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:33.958931 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:33.958947 8726 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:11:33.959106 8726 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:11:33.959540 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:11:33.959785 8726 catalog_manager.cc:1357] Generated new cluster ID: f7b3c11dcbc3470d9a05b09cd8f4c088
I20251026 09:11:33.959839 8726 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:11:33.972260 8726 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:11:33.972731 8726 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:11:33.985548 8726 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129: Generated new TSK 0
I20251026 09:11:33.985666 8726 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:11:33.991608 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:33.993211 8737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:33.993368 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:33.993311 8738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:33.993340 8741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:33.993539 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:33.993578 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:33.993592 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469893993593 us; error 0 us; skew 500 ppm
I20251026 09:11:33.994267 3128 webserver.cc:492] Webserver started at http://127.3.14.1:46123/ using document root <none> and password file <none>
I20251026 09:11:33.994349 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:33.994391 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:33.994452 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:33.994746 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe"
format_stamp: "Formatted at 2025-10-26 09:11:33 on dist-test-slave-w8k0"
I20251026 09:11:33.995726 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:33.996274 8745 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:33.996379 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:33.996431 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe"
format_stamp: "Formatted at 2025-10-26 09:11:33 on dist-test-slave-w8k0"
I20251026 09:11:33.996479 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:34.008992 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:34.009286 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:34.009457 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:34.009677 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:34.009709 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:34.009744 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:34.009775 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:34.012696 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:37621
I20251026 09:11:34.012738 8808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:37621 every 8 connection(s)
I20251026 09:11:34.013129 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:34.014453 8814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:34.014475 8813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:34.014768 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:34.014776 8816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:34.014963 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:34.014997 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:34.015019 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469894015019 us; error 0 us; skew 500 ppm
I20251026 09:11:34.015892 3128 webserver.cc:492] Webserver started at http://127.3.14.2:46087/ using document root <none> and password file <none>
I20251026 09:11:34.016003 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:34.016077 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:34.016180 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:34.016530 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "a9b8809cbf5b418eb162e9236454ce45"
format_stamp: "Formatted at 2025-10-26 09:11:34 on dist-test-slave-w8k0"
I20251026 09:11:34.017544 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:34.018127 8821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:34.018304 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:34.018358 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "a9b8809cbf5b418eb162e9236454ce45"
format_stamp: "Formatted at 2025-10-26 09:11:34 on dist-test-slave-w8k0"
I20251026 09:11:34.018412 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:34.021569 8809 heartbeater.cc:344] Connected to a master server at 127.3.14.62:41575
I20251026 09:11:34.021653 8809 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:34.021787 8809 heartbeater.cc:507] Master 127.3.14.62:41575 requested a full tablet report, sending...
I20251026 09:11:34.022068 8680 ts_manager.cc:194] Registered new tserver with Master: 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621)
I20251026 09:11:34.022650 8680 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48346
I20251026 09:11:34.026752 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:34.027022 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:34.027201 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:34.027454 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:34.027486 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:34.027519 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:34.027546 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:34.030344 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:42977
I20251026 09:11:34.030421 8884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:42977 every 8 connection(s)
I20251026 09:11:34.030772 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:34.032131 8891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:34.032234 8888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:34.032395 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:34.032752 8889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:34.032927 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:34.032979 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:34.032997 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469894032998 us; error 0 us; skew 500 ppm
I20251026 09:11:34.033933 3128 webserver.cc:492] Webserver started at http://127.3.14.3:35823/ using document root <none> and password file <none>
I20251026 09:11:34.034035 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:34.034101 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:34.034147 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:34.034246 8885 heartbeater.cc:344] Connected to a master server at 127.3.14.62:41575
I20251026 09:11:34.034307 8885 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:34.034397 8885 heartbeater.cc:507] Master 127.3.14.62:41575 requested a full tablet report, sending...
I20251026 09:11:34.034495 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "44797ee52d314ac6a8032d536eb19e73"
format_stamp: "Formatted at 2025-10-26 09:11:34 on dist-test-slave-w8k0"
I20251026 09:11:34.034648 8680 ts_manager.cc:194] Registered new tserver with Master: a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2:42977)
I20251026 09:11:34.034974 8680 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48354
I20251026 09:11:34.035583 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 09:11:34.036114 8896 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:34.036350 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:34.036414 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "44797ee52d314ac6a8032d536eb19e73"
format_stamp: "Formatted at 2025-10-26 09:11:34 on dist-test-slave-w8k0"
I20251026 09:11:34.036475 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:34.040122 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:34.040390 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:34.040560 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:34.040768 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:34.040802 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:34.040838 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:34.040864 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:34.044353 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:41349
I20251026 09:11:34.044889 8959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:41349 every 8 connection(s)
I20251026 09:11:34.049443 8960 heartbeater.cc:344] Connected to a master server at 127.3.14.62:41575
I20251026 09:11:34.049532 8960 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:34.049686 8960 heartbeater.cc:507] Master 127.3.14.62:41575 requested a full tablet report, sending...
I20251026 09:11:34.050011 8680 ts_manager.cc:194] Registered new tserver with Master: 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349)
I20251026 09:11:34.050086 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005539626s
I20251026 09:11:34.050537 8680 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48368
I20251026 09:11:35.023257 8809 heartbeater.cc:499] Master 127.3.14.62:41575 was elected leader, sending a full tablet report...
I20251026 09:11:35.035575 8885 heartbeater.cc:499] Master 127.3.14.62:41575 was elected leader, sending a full tablet report...
I20251026 09:11:35.051133 8960 heartbeater.cc:499] Master 127.3.14.62:41575 was elected leader, sending a full tablet report...
I20251026 09:11:35.075820 3128 test_util.cc:276] Using random seed: 727771112
I20251026 09:11:35.082381 8680 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48378:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:11:35.082574 8680 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:11:35.088541 8924 tablet_service.cc:1505] Processing CreateTablet for tablet 95b88dca28844b6a991216145224b07e (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:11:35.088783 8924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b88dca28844b6a991216145224b07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.089239 8849 tablet_service.cc:1505] Processing CreateTablet for tablet 95b88dca28844b6a991216145224b07e (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:11:35.089265 8848 tablet_service.cc:1505] Processing CreateTablet for tablet 26e16f5cbf2040f1bdced2c676615707 (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:11:35.089393 8849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b88dca28844b6a991216145224b07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.089569 8848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e16f5cbf2040f1bdced2c676615707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.090443 8980 tablet_bootstrap.cc:492] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73: Bootstrap starting.
I20251026 09:11:35.090592 8921 tablet_service.cc:1505] Processing CreateTablet for tablet c5aeba37843040c5a4bb0422bedd2513 (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:11:35.090677 8921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5aeba37843040c5a4bb0422bedd2513. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.091001 8980 tablet_bootstrap.cc:654] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.091099 8847 tablet_service.cc:1505] Processing CreateTablet for tablet c761acf6dd7d43b08bcb48d9092e1881 (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:11:35.091216 8847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c761acf6dd7d43b08bcb48d9092e1881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.091706 8980 tablet_bootstrap.cc:492] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73: No bootstrap required, opened a new log
I20251026 09:11:35.091751 8980 ts_tablet_manager.cc:1403] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:35.091928 8980 raft_consensus.cc:359] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } }
I20251026 09:11:35.092001 8980 raft_consensus.cc:385] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.092023 8980 raft_consensus.cc:740] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44797ee52d314ac6a8032d536eb19e73, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.092085 8980 consensus_queue.cc:260] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } }
I20251026 09:11:35.092283 8980 ts_tablet_manager.cc:1434] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:35.092451 8980 tablet_bootstrap.cc:492] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73: Bootstrap starting.
I20251026 09:11:35.092417 8846 tablet_service.cc:1505] Processing CreateTablet for tablet c5aeba37843040c5a4bb0422bedd2513 (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:11:35.092509 8773 tablet_service.cc:1505] Processing CreateTablet for tablet 95b88dca28844b6a991216145224b07e (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:11:35.092586 8846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5aeba37843040c5a4bb0422bedd2513. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.092607 8773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b88dca28844b6a991216145224b07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.092857 8981 tablet_bootstrap.cc:492] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45: Bootstrap starting.
I20251026 09:11:35.092929 8980 tablet_bootstrap.cc:654] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.093353 8981 tablet_bootstrap.cc:654] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.094081 8981 tablet_bootstrap.cc:492] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45: No bootstrap required, opened a new log
I20251026 09:11:35.094133 8981 ts_tablet_manager.cc:1403] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:35.094126 8986 tablet_bootstrap.cc:492] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe: Bootstrap starting.
I20251026 09:11:35.094187 8772 tablet_service.cc:1505] Processing CreateTablet for tablet 26e16f5cbf2040f1bdced2c676615707 (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:11:35.094300 8772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e16f5cbf2040f1bdced2c676615707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.094472 8981 raft_consensus.cc:359] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } }
I20251026 09:11:35.092412 8770 tablet_service.cc:1505] Processing CreateTablet for tablet c5aeba37843040c5a4bb0422bedd2513 (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:11:35.094559 8981 raft_consensus.cc:385] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.094583 8981 raft_consensus.cc:740] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b8809cbf5b418eb162e9236454ce45, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.094650 8986 tablet_bootstrap.cc:654] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.094659 8981 consensus_queue.cc:260] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } }
I20251026 09:11:35.094640 8770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5aeba37843040c5a4bb0422bedd2513. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.094846 8981 ts_tablet_manager.cc:1434] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:35.095243 8986 tablet_bootstrap.cc:492] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe: No bootstrap required, opened a new log
I20251026 09:11:35.095284 8986 ts_tablet_manager.cc:1403] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:35.095395 8981 tablet_bootstrap.cc:492] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45: Bootstrap starting.
I20251026 09:11:35.090502 8922 tablet_service.cc:1505] Processing CreateTablet for tablet c761acf6dd7d43b08bcb48d9092e1881 (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:11:35.096256 8771 tablet_service.cc:1505] Processing CreateTablet for tablet c761acf6dd7d43b08bcb48d9092e1881 (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:11:35.096385 8771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c761acf6dd7d43b08bcb48d9092e1881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.090454 8923 tablet_service.cc:1505] Processing CreateTablet for tablet 26e16f5cbf2040f1bdced2c676615707 (DEFAULT_TABLE table=test-workload [id=7c377061a3b74f8ab92ded933aea56fc]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:11:35.096810 8922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c761acf6dd7d43b08bcb48d9092e1881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.097548 8981 tablet_bootstrap.cc:654] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.096261 8986 raft_consensus.cc:359] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } }
I20251026 09:11:35.097740 8986 raft_consensus.cc:385] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.097765 8986 raft_consensus.cc:740] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc7a7a32bff4c96b276e4b495ae3fbe, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.097803 8986 consensus_queue.cc:260] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } }
I20251026 09:11:35.098001 8986 ts_tablet_manager.cc:1434] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251026 09:11:35.098084 8986 tablet_bootstrap.cc:492] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Bootstrap starting.
I20251026 09:11:35.098275 8923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e16f5cbf2040f1bdced2c676615707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:35.098526 8986 tablet_bootstrap.cc:654] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.099086 8986 tablet_bootstrap.cc:492] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe: No bootstrap required, opened a new log
I20251026 09:11:35.099130 8986 ts_tablet_manager.cc:1403] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:35.099277 8986 raft_consensus.cc:359] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.099375 8986 raft_consensus.cc:385] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.099401 8986 raft_consensus.cc:740] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc7a7a32bff4c96b276e4b495ae3fbe, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.099449 8986 consensus_queue.cc:260] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.099542 8986 ts_tablet_manager.cc:1434] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:35.099596 8986 tablet_bootstrap.cc:492] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Bootstrap starting.
I20251026 09:11:35.099835 8980 tablet_bootstrap.cc:492] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73: No bootstrap required, opened a new log
I20251026 09:11:35.099881 8980 ts_tablet_manager.cc:1403] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20251026 09:11:35.100028 8986 tablet_bootstrap.cc:654] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.100052 8980 raft_consensus.cc:359] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.100118 8980 raft_consensus.cc:385] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.100140 8980 raft_consensus.cc:740] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44797ee52d314ac6a8032d536eb19e73, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.100190 8980 consensus_queue.cc:260] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.100278 8980 ts_tablet_manager.cc:1434] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:35.100337 8980 tablet_bootstrap.cc:492] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73: Bootstrap starting.
I20251026 09:11:35.100785 8980 tablet_bootstrap.cc:654] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.100845 8986 tablet_bootstrap.cc:492] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe: No bootstrap required, opened a new log
I20251026 09:11:35.100898 8986 ts_tablet_manager.cc:1403] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:35.101042 8986 raft_consensus.cc:359] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.101094 8986 raft_consensus.cc:385] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.101107 8986 raft_consensus.cc:740] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc7a7a32bff4c96b276e4b495ae3fbe, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.101222 8986 consensus_queue.cc:260] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.101317 8981 tablet_bootstrap.cc:492] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45: No bootstrap required, opened a new log
I20251026 09:11:35.101364 8981 ts_tablet_manager.cc:1403] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20251026 09:11:35.101408 8980 tablet_bootstrap.cc:492] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73: No bootstrap required, opened a new log
I20251026 09:11:35.101454 8980 ts_tablet_manager.cc:1403] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:35.101480 8986 ts_tablet_manager.cc:1434] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:11:35.101526 8981 raft_consensus.cc:359] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.101579 8980 raft_consensus.cc:359] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.101603 8981 raft_consensus.cc:385] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.101549 8986 tablet_bootstrap.cc:492] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Bootstrap starting.
I20251026 09:11:35.101660 8980 raft_consensus.cc:385] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.101665 8981 raft_consensus.cc:740] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b8809cbf5b418eb162e9236454ce45, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.101694 8980 raft_consensus.cc:740] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44797ee52d314ac6a8032d536eb19e73, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.101739 8981 consensus_queue.cc:260] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.101742 8980 consensus_queue.cc:260] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.101934 8980 ts_tablet_manager.cc:1434] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:35.101995 8980 tablet_bootstrap.cc:492] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73: Bootstrap starting.
I20251026 09:11:35.102228 8986 tablet_bootstrap.cc:654] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.102382 8980 tablet_bootstrap.cc:654] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.101840 8981 ts_tablet_manager.cc:1434] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:35.102819 8986 tablet_bootstrap.cc:492] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe: No bootstrap required, opened a new log
I20251026 09:11:35.102866 8986 ts_tablet_manager.cc:1403] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:11:35.102885 8981 tablet_bootstrap.cc:492] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45: Bootstrap starting.
I20251026 09:11:35.103133 8980 tablet_bootstrap.cc:492] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73: No bootstrap required, opened a new log
I20251026 09:11:35.103164 8980 ts_tablet_manager.cc:1403] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:35.103308 8980 raft_consensus.cc:359] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.103395 8981 tablet_bootstrap.cc:654] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.103400 8980 raft_consensus.cc:385] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.103453 8980 raft_consensus.cc:740] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44797ee52d314ac6a8032d536eb19e73, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.103502 8980 consensus_queue.cc:260] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.103639 8980 ts_tablet_manager.cc:1434] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:35.104076 8981 tablet_bootstrap.cc:492] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45: No bootstrap required, opened a new log
I20251026 09:11:35.104038 8986 raft_consensus.cc:359] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.104120 8986 raft_consensus.cc:385] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.104164 8986 raft_consensus.cc:740] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc7a7a32bff4c96b276e4b495ae3fbe, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.104128 8981 ts_tablet_manager.cc:1403] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:35.104239 8986 consensus_queue.cc:260] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.104384 8986 ts_tablet_manager.cc:1434] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:35.104415 8981 raft_consensus.cc:359] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.104475 8981 raft_consensus.cc:385] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.104504 8981 raft_consensus.cc:740] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b8809cbf5b418eb162e9236454ce45, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.104550 8981 consensus_queue.cc:260] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.104630 8981 ts_tablet_manager.cc:1434] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:35.104696 8981 tablet_bootstrap.cc:492] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45: Bootstrap starting.
I20251026 09:11:35.105080 8981 tablet_bootstrap.cc:654] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:35.105676 8981 tablet_bootstrap.cc:492] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45: No bootstrap required, opened a new log
I20251026 09:11:35.105736 8981 ts_tablet_manager.cc:1403] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:35.105908 8981 raft_consensus.cc:359] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.105974 8981 raft_consensus.cc:385] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:35.105996 8981 raft_consensus.cc:740] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b8809cbf5b418eb162e9236454ce45, State: Initialized, Role: FOLLOWER
I20251026 09:11:35.106047 8981 consensus_queue.cc:260] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.106158 8981 ts_tablet_manager.cc:1434] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:35.194136 8987 raft_consensus.cc:493] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:35.194265 8987 raft_consensus.cc:515] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } }
I20251026 09:11:35.194479 8987 leader_election.cc:290] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349), 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621)
I20251026 09:11:35.197602 8783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95b88dca28844b6a991216145224b07e" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" is_pre_election: true
I20251026 09:11:35.197739 8783 raft_consensus.cc:2468] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 0.
I20251026 09:11:35.197716 8934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95b88dca28844b6a991216145224b07e" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44797ee52d314ac6a8032d536eb19e73" is_pre_election: true
I20251026 09:11:35.197852 8934 raft_consensus.cc:2468] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 0.
I20251026 09:11:35.197942 8824 leader_election.cc:304] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc7a7a32bff4c96b276e4b495ae3fbe, a9b8809cbf5b418eb162e9236454ce45; no voters:
I20251026 09:11:35.198042 8987 raft_consensus.cc:2804] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:35.198096 8987 raft_consensus.cc:493] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:35.198125 8987 raft_consensus.cc:3060] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.198814 8987 raft_consensus.cc:515] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } }
I20251026 09:11:35.198987 8987 leader_election.cc:290] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 1 election: Requested vote from peers 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349), 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621)
I20251026 09:11:35.199270 8934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95b88dca28844b6a991216145224b07e" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44797ee52d314ac6a8032d536eb19e73"
I20251026 09:11:35.199270 8783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95b88dca28844b6a991216145224b07e" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe"
I20251026 09:11:35.199432 8934 raft_consensus.cc:3060] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.199435 8783 raft_consensus.cc:3060] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.200264 8783 raft_consensus.cc:2468] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 1.
I20251026 09:11:35.200264 8934 raft_consensus.cc:2468] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 1.
I20251026 09:11:35.200488 8824 leader_election.cc:304] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc7a7a32bff4c96b276e4b495ae3fbe, a9b8809cbf5b418eb162e9236454ce45; no voters:
I20251026 09:11:35.200525 8983 raft_consensus.cc:493] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:35.200611 8987 raft_consensus.cc:2804] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:35.200598 8983 raft_consensus.cc:515] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.200770 8983 leader_election.cc:290] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621), a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2:42977)
I20251026 09:11:35.200778 8987 raft_consensus.cc:697] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Becoming Leader. State: Replica: a9b8809cbf5b418eb162e9236454ce45, State: Running, Role: LEADER
I20251026 09:11:35.200899 8987 consensus_queue.cc:237] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } }
I20251026 09:11:35.201718 8680 catalog_manager.cc:5649] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } }
I20251026 09:11:35.204226 8783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26e16f5cbf2040f1bdced2c676615707" candidate_uuid: "44797ee52d314ac6a8032d536eb19e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" is_pre_election: true
I20251026 09:11:35.204330 8859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26e16f5cbf2040f1bdced2c676615707" candidate_uuid: "44797ee52d314ac6a8032d536eb19e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b8809cbf5b418eb162e9236454ce45" is_pre_election: true
I20251026 09:11:35.204337 8783 raft_consensus.cc:2468] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44797ee52d314ac6a8032d536eb19e73 in term 0.
I20251026 09:11:35.204407 8859 raft_consensus.cc:2468] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44797ee52d314ac6a8032d536eb19e73 in term 0.
I20251026 09:11:35.204531 8899 leader_election.cc:304] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc7a7a32bff4c96b276e4b495ae3fbe, 44797ee52d314ac6a8032d536eb19e73; no voters:
I20251026 09:11:35.204643 8983 raft_consensus.cc:2804] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:35.204713 8983 raft_consensus.cc:493] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:35.204733 8983 raft_consensus.cc:3060] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.205431 8983 raft_consensus.cc:515] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.205574 8983 leader_election.cc:290] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [CANDIDATE]: Term 1 election: Requested vote from peers 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621), a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2:42977)
I20251026 09:11:35.205760 8783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26e16f5cbf2040f1bdced2c676615707" candidate_uuid: "44797ee52d314ac6a8032d536eb19e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe"
I20251026 09:11:35.205782 8859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26e16f5cbf2040f1bdced2c676615707" candidate_uuid: "44797ee52d314ac6a8032d536eb19e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b8809cbf5b418eb162e9236454ce45"
I20251026 09:11:35.205850 8783 raft_consensus.cc:3060] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.205864 8859 raft_consensus.cc:3060] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.206544 8859 raft_consensus.cc:2468] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44797ee52d314ac6a8032d536eb19e73 in term 1.
I20251026 09:11:35.206550 8783 raft_consensus.cc:2468] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44797ee52d314ac6a8032d536eb19e73 in term 1.
I20251026 09:11:35.206744 8899 leader_election.cc:304] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc7a7a32bff4c96b276e4b495ae3fbe, 44797ee52d314ac6a8032d536eb19e73; no voters:
I20251026 09:11:35.206861 8983 raft_consensus.cc:2804] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:35.207051 8983 raft_consensus.cc:697] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 1 LEADER]: Becoming Leader. State: Replica: 44797ee52d314ac6a8032d536eb19e73, State: Running, Role: LEADER
I20251026 09:11:35.207119 8983 consensus_queue.cc:237] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.207828 8680 catalog_manager.cc:5649] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3). New cstate: current_term: 1 leader_uuid: "44797ee52d314ac6a8032d536eb19e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: HEALTHY } } }
I20251026 09:11:35.222755 8990 raft_consensus.cc:493] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:35.222898 8990 raft_consensus.cc:515] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.223129 8990 leader_election.cc:290] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2:42977), 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349)
I20251026 09:11:35.226081 8859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c761acf6dd7d43b08bcb48d9092e1881" candidate_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b8809cbf5b418eb162e9236454ce45" is_pre_election: true
I20251026 09:11:35.226223 8859 raft_consensus.cc:2468] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc7a7a32bff4c96b276e4b495ae3fbe in term 0.
I20251026 09:11:35.226081 8934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c761acf6dd7d43b08bcb48d9092e1881" candidate_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44797ee52d314ac6a8032d536eb19e73" is_pre_election: true
I20251026 09:11:35.226287 8934 raft_consensus.cc:2468] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc7a7a32bff4c96b276e4b495ae3fbe in term 0.
I20251026 09:11:35.226410 8747 leader_election.cc:304] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc7a7a32bff4c96b276e4b495ae3fbe, a9b8809cbf5b418eb162e9236454ce45; no voters:
I20251026 09:11:35.226531 8990 raft_consensus.cc:2804] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:35.226586 8990 raft_consensus.cc:493] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:35.226610 8990 raft_consensus.cc:3060] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.227389 8990 raft_consensus.cc:515] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.227541 8990 leader_election.cc:290] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [CANDIDATE]: Term 1 election: Requested vote from peers a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2:42977), 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349)
I20251026 09:11:35.227681 8934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c761acf6dd7d43b08bcb48d9092e1881" candidate_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44797ee52d314ac6a8032d536eb19e73"
I20251026 09:11:35.227715 8859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c761acf6dd7d43b08bcb48d9092e1881" candidate_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b8809cbf5b418eb162e9236454ce45"
I20251026 09:11:35.227753 8934 raft_consensus.cc:3060] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.227780 8859 raft_consensus.cc:3060] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.228406 8859 raft_consensus.cc:2468] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc7a7a32bff4c96b276e4b495ae3fbe in term 1.
I20251026 09:11:35.228406 8934 raft_consensus.cc:2468] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc7a7a32bff4c96b276e4b495ae3fbe in term 1.
I20251026 09:11:35.228610 8749 leader_election.cc:304] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc7a7a32bff4c96b276e4b495ae3fbe, 44797ee52d314ac6a8032d536eb19e73; no voters:
I20251026 09:11:35.228741 8990 raft_consensus.cc:2804] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:35.228943 8990 raft_consensus.cc:697] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 LEADER]: Becoming Leader. State: Replica: 2cc7a7a32bff4c96b276e4b495ae3fbe, State: Running, Role: LEADER
I20251026 09:11:35.229014 8990 consensus_queue.cc:237] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.229610 8680 catalog_manager.cc:5649] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1). New cstate: current_term: 1 leader_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:35.273696 8987 raft_consensus.cc:493] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:35.273837 8987 raft_consensus.cc:515] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.274022 8987 leader_election.cc:290] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621), 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349)
I20251026 09:11:35.274309 8783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5aeba37843040c5a4bb0422bedd2513" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" is_pre_election: true
I20251026 09:11:35.274403 8783 raft_consensus.cc:2468] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 0.
I20251026 09:11:35.274312 8934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5aeba37843040c5a4bb0422bedd2513" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44797ee52d314ac6a8032d536eb19e73" is_pre_election: true
I20251026 09:11:35.274503 8934 raft_consensus.cc:2468] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 0.
I20251026 09:11:35.274595 8824 leader_election.cc:304] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc7a7a32bff4c96b276e4b495ae3fbe, a9b8809cbf5b418eb162e9236454ce45; no voters:
I20251026 09:11:35.274699 8987 raft_consensus.cc:2804] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:35.274737 8987 raft_consensus.cc:493] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:35.274770 8987 raft_consensus.cc:3060] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.275537 8987 raft_consensus.cc:515] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.275692 8987 leader_election.cc:290] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 1 election: Requested vote from peers 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621), 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349)
I20251026 09:11:35.275889 8783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5aeba37843040c5a4bb0422bedd2513" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe"
I20251026 09:11:35.275889 8934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5aeba37843040c5a4bb0422bedd2513" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44797ee52d314ac6a8032d536eb19e73"
I20251026 09:11:35.276005 8783 raft_consensus.cc:3060] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.276013 8934 raft_consensus.cc:3060] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:35.276772 8934 raft_consensus.cc:2468] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 1.
I20251026 09:11:35.276772 8783 raft_consensus.cc:2468] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 1.
I20251026 09:11:35.277027 8824 leader_election.cc:304] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc7a7a32bff4c96b276e4b495ae3fbe, a9b8809cbf5b418eb162e9236454ce45; no voters:
I20251026 09:11:35.277115 8987 raft_consensus.cc:2804] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:35.277200 8987 raft_consensus.cc:697] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Becoming Leader. State: Replica: a9b8809cbf5b418eb162e9236454ce45, State: Running, Role: LEADER
I20251026 09:11:35.277272 8987 consensus_queue.cc:237] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:35.277992 8680 catalog_manager.cc:5649] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:35.313408 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:11:35.316516 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:35.316757 3128 tablet_replica.cc:333] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe: stopping tablet replica
I20251026 09:11:35.316849 3128 raft_consensus.cc:2243] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:35.316907 3128 pending_rounds.cc:70] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Trying to abort 1 pending ops.
I20251026 09:11:35.316927 3128 pending_rounds.cc:77] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214980690858659840 op_type: NO_OP noop_request { }
I20251026 09:11:35.316977 3128 raft_consensus.cc:2889] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 09:11:35.317005 3128 raft_consensus.cc:2272] T c761acf6dd7d43b08bcb48d9092e1881 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:35.317322 3128 tablet_replica.cc:333] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe: stopping tablet replica
I20251026 09:11:35.317369 3128 raft_consensus.cc:2243] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:35.317394 3128 raft_consensus.cc:2272] T c5aeba37843040c5a4bb0422bedd2513 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:35.317518 3128 tablet_replica.cc:333] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe: stopping tablet replica
I20251026 09:11:35.317548 3128 raft_consensus.cc:2243] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:35.317574 3128 raft_consensus.cc:2272] T 95b88dca28844b6a991216145224b07e P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:35.317759 3128 tablet_replica.cc:333] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe: stopping tablet replica
I20251026 09:11:35.317802 3128 raft_consensus.cc:2243] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:35.317824 3128 raft_consensus.cc:2272] T 26e16f5cbf2040f1bdced2c676615707 P 2cc7a7a32bff4c96b276e4b495ae3fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:35.330225 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
W20251026 09:11:35.585861 8899 consensus_peers.cc:597] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:35.639422 8987 consensus_queue.cc:1048] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:11:35.641408 8824 consensus_peers.cc:597] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:35.753692 8983 consensus_queue.cc:1048] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:11:35.821450 8824 consensus_peers.cc:597] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:35.871604 9014 consensus_queue.cc:1048] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:36.331871 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:36.333230 9022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:36.333298 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:36.333453 9023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:36.333453 9025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:36.333678 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:36.333736 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:36.333758 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469896333759 us; error 0 us; skew 500 ppm
I20251026 09:11:36.334452 3128 webserver.cc:492] Webserver started at http://127.3.14.4:45877/ using document root <none> and password file <none>
I20251026 09:11:36.334546 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:36.334604 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:36.334656 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:36.334992 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-3-root/instance:
uuid: "68b888029fc5492c8246f381b8d634a1"
format_stamp: "Formatted at 2025-10-26 09:11:36 on dist-test-slave-w8k0"
I20251026 09:11:36.335973 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:36.336454 9030 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:36.336589 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:36.336625 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-3-root
uuid: "68b888029fc5492c8246f381b8d634a1"
format_stamp: "Formatted at 2025-10-26 09:11:36 on dist-test-slave-w8k0"
I20251026 09:11:36.336674 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761469742341867-3128-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:36.353096 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:36.353323 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:36.353502 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:36.353703 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:36.353732 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:36.353767 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:36.353798 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:36.356827 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.4:46557
I20251026 09:11:36.356918 9093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.4:46557 every 8 connection(s)
I20251026 09:11:36.360980 9094 heartbeater.cc:344] Connected to a master server at 127.3.14.62:41575
I20251026 09:11:36.361083 9094 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:36.361212 9094 heartbeater.cc:507] Master 127.3.14.62:41575 requested a full tablet report, sending...
I20251026 09:11:36.361441 8679 ts_manager.cc:194] Registered new tserver with Master: 68b888029fc5492c8246f381b8d634a1 (127.3.14.4:46557)
I20251026 09:11:36.361838 8679 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48390
I20251026 09:11:36.583432 9014 consensus_queue.cc:579] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Leader has been unable to successfully communicate with peer 2cc7a7a32bff4c96b276e4b495ae3fbe for more than 1 seconds (1.382s)
W20251026 09:11:36.584017 8824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20251026 09:11:36.584568 8859 consensus_queue.cc:237] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:36.585130 8934 raft_consensus.cc:1275] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251026 09:11:36.585588 8824 consensus_peers.cc:597] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:36.585625 9098 consensus_queue.cc:1048] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:36.586431 9014 raft_consensus.cc:2955] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } } }
I20251026 09:11:36.587478 8934 raft_consensus.cc:2955] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } } }
I20251026 09:11:36.587808 8665 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 95b88dca28844b6a991216145224b07e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251026 09:11:36.587960 8679 catalog_manager.cc:5649] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 reported cstate change: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 09:11:36.589475 8822 consensus_peers.cc:597] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 -> Peer 68b888029fc5492c8246f381b8d634a1 (127.3.14.4:46557): Couldn't send request to peer 68b888029fc5492c8246f381b8d634a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95b88dca28844b6a991216145224b07e. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:36.669245 9018 consensus_queue.cc:579] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Leader has been unable to successfully communicate with peer 2cc7a7a32bff4c96b276e4b495ae3fbe for more than 1 seconds (1.462s)
I20251026 09:11:36.670359 8934 consensus_queue.cc:237] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:36.670939 8859 raft_consensus.cc:1275] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Refusing update from remote peer 44797ee52d314ac6a8032d536eb19e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:11:36.671221 9107 consensus_queue.cc:1048] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 09:11:36.671345 8899 consensus_peers.cc:597] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:36.672004 9018 raft_consensus.cc:2955] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } } }
I20251026 09:11:36.673229 8859 raft_consensus.cc:2955] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } } }
I20251026 09:11:36.673368 8667 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 26e16f5cbf2040f1bdced2c676615707 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251026 09:11:36.673614 8680 catalog_manager.cc:5649] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 reported cstate change: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "44797ee52d314ac6a8032d536eb19e73" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 09:11:36.673943 8897 consensus_peers.cc:597] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 -> Peer 68b888029fc5492c8246f381b8d634a1 (127.3.14.4:46557): Couldn't send request to peer 68b888029fc5492c8246f381b8d634a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26e16f5cbf2040f1bdced2c676615707. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:36.749003 9098 raft_consensus.cc:493] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:36.749143 9098 raft_consensus.cc:515] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:36.749369 9098 leader_election.cc:290] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621), 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349)
I20251026 09:11:36.749620 8934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c761acf6dd7d43b08bcb48d9092e1881" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44797ee52d314ac6a8032d536eb19e73" is_pre_election: true
I20251026 09:11:36.749696 8934 raft_consensus.cc:2468] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 1.
I20251026 09:11:36.749879 8825 leader_election.cc:304] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44797ee52d314ac6a8032d536eb19e73, a9b8809cbf5b418eb162e9236454ce45; no voters:
I20251026 09:11:36.750026 9098 raft_consensus.cc:2804] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251026 09:11:36.750061 8824 leader_election.cc:336] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111)
I20251026 09:11:36.750077 9098 raft_consensus.cc:493] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:36.750138 9098 raft_consensus.cc:3060] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:36.750880 9098 raft_consensus.cc:515] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:36.751044 9098 leader_election.cc:290] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 2 election: Requested vote from peers 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621), 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349)
I20251026 09:11:36.751250 8934 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c761acf6dd7d43b08bcb48d9092e1881" candidate_uuid: "a9b8809cbf5b418eb162e9236454ce45" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44797ee52d314ac6a8032d536eb19e73"
I20251026 09:11:36.751358 8934 raft_consensus.cc:3060] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Advancing to term 2
W20251026 09:11:36.751644 8824 leader_election.cc:336] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111)
I20251026 09:11:36.751972 8934 raft_consensus.cc:2468] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9b8809cbf5b418eb162e9236454ce45 in term 2.
I20251026 09:11:36.752159 8825 leader_election.cc:304] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44797ee52d314ac6a8032d536eb19e73, a9b8809cbf5b418eb162e9236454ce45; no voters: 2cc7a7a32bff4c96b276e4b495ae3fbe
I20251026 09:11:36.752243 9098 raft_consensus.cc:2804] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:11:36.752297 9098 raft_consensus.cc:697] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 2 LEADER]: Becoming Leader. State: Replica: a9b8809cbf5b418eb162e9236454ce45, State: Running, Role: LEADER
I20251026 09:11:36.752350 9098 consensus_queue.cc:237] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } }
I20251026 09:11:36.752998 8679 catalog_manager.cc:5649] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 reported cstate change: term changed from 1 to 2, leader changed from 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) to a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2). New cstate: current_term: 2 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:36.807384 9098 consensus_queue.cc:579] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Leader has been unable to successfully communicate with peer 2cc7a7a32bff4c96b276e4b495ae3fbe for more than 1 seconds (1.530s)
I20251026 09:11:36.808466 8859 consensus_queue.cc:237] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:36.808987 8934 raft_consensus.cc:1275] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251026 09:11:36.809084 8822 consensus_peers.cc:597] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 -> Peer 68b888029fc5492c8246f381b8d634a1 (127.3.14.4:46557): Couldn't send request to peer 68b888029fc5492c8246f381b8d634a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5aeba37843040c5a4bb0422bedd2513. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:36.809257 9014 consensus_queue.cc:1048] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 09:11:36.809345 8824 consensus_peers.cc:597] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:36.810060 9098 raft_consensus.cc:2955] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } } }
I20251026 09:11:36.810237 8934 raft_consensus.cc:2955] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } } }
I20251026 09:11:36.811141 8665 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c5aeba37843040c5a4bb0422bedd2513 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251026 09:11:36.811267 8680 catalog_manager.cc:5649] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 reported cstate change: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 09:11:36.959806 8728 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 2cc7a7a32bff4c96b276e4b495ae3fbe not available for placement
I20251026 09:11:37.068903 9111 ts_tablet_manager.cc:933] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: Initiating tablet copy from peer 44797ee52d314ac6a8032d536eb19e73 (127.3.14.3:41349)
I20251026 09:11:37.069105 9111 tablet_copy_client.cc:323] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:41349
I20251026 09:11:37.071815 8944 tablet_copy_service.cc:140] P 44797ee52d314ac6a8032d536eb19e73: Received BeginTabletCopySession request for tablet 26e16f5cbf2040f1bdced2c676615707 from peer 68b888029fc5492c8246f381b8d634a1 ({username='slave'} at 127.0.0.1:41490)
I20251026 09:11:37.071902 8944 tablet_copy_service.cc:161] P 44797ee52d314ac6a8032d536eb19e73: Beginning new tablet copy session on tablet 26e16f5cbf2040f1bdced2c676615707 from peer 68b888029fc5492c8246f381b8d634a1 at {username='slave'} at 127.0.0.1:41490: session id = 68b888029fc5492c8246f381b8d634a1-26e16f5cbf2040f1bdced2c676615707
I20251026 09:11:37.072432 8944 tablet_copy_source_session.cc:215] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:11:37.072841 9111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e16f5cbf2040f1bdced2c676615707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:37.074008 9111 tablet_copy_client.cc:806] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Starting download of 0 data blocks...
I20251026 09:11:37.074154 9111 tablet_copy_client.cc:670] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Starting download of 1 WAL segments...
I20251026 09:11:37.074779 9111 tablet_copy_client.cc:538] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:37.075515 9111 tablet_bootstrap.cc:492] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: Bootstrap starting.
I20251026 09:11:37.076838 9111 tablet_bootstrap.cc:492] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:37.077068 9111 tablet_bootstrap.cc:492] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: Bootstrap complete.
I20251026 09:11:37.077183 9111 ts_tablet_manager.cc:1403] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:11:37.077313 9111 raft_consensus.cc:359] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:37.077384 9111 raft_consensus.cc:740] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 68b888029fc5492c8246f381b8d634a1, State: Initialized, Role: LEARNER
I20251026 09:11:37.077410 9111 consensus_queue.cc:260] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:37.077585 9111 ts_tablet_manager.cc:1434] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:37.077647 9094 heartbeater.cc:499] Master 127.3.14.62:41575 was elected leader, sending a full tablet report...
I20251026 09:11:37.077881 8944 tablet_copy_service.cc:342] P 44797ee52d314ac6a8032d536eb19e73: Request end of tablet copy session 68b888029fc5492c8246f381b8d634a1-26e16f5cbf2040f1bdced2c676615707 received from {username='slave'} at 127.0.0.1:41490
I20251026 09:11:37.077940 8944 tablet_copy_service.cc:434] P 44797ee52d314ac6a8032d536eb19e73: ending tablet copy session 68b888029fc5492c8246f381b8d634a1-26e16f5cbf2040f1bdced2c676615707 on tablet 26e16f5cbf2040f1bdced2c676615707 with peer 68b888029fc5492c8246f381b8d634a1
I20251026 09:11:37.159420 8934 raft_consensus.cc:1275] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 09:11:37.159786 9110 consensus_queue.cc:1048] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:11:37.161597 8824 consensus_peers.cc:597] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:37.200474 9111 ts_tablet_manager.cc:933] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: Initiating tablet copy from peer a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2:42977)
I20251026 09:11:37.200685 9111 tablet_copy_client.cc:323] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42977
I20251026 09:11:37.203303 8869 tablet_copy_service.cc:140] P a9b8809cbf5b418eb162e9236454ce45: Received BeginTabletCopySession request for tablet 95b88dca28844b6a991216145224b07e from peer 68b888029fc5492c8246f381b8d634a1 ({username='slave'} at 127.0.0.1:37728)
I20251026 09:11:37.203450 8869 tablet_copy_service.cc:161] P a9b8809cbf5b418eb162e9236454ce45: Beginning new tablet copy session on tablet 95b88dca28844b6a991216145224b07e from peer 68b888029fc5492c8246f381b8d634a1 at {username='slave'} at 127.0.0.1:37728: session id = 68b888029fc5492c8246f381b8d634a1-95b88dca28844b6a991216145224b07e
I20251026 09:11:37.203972 8869 tablet_copy_source_session.cc:215] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:11:37.204367 9111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b88dca28844b6a991216145224b07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:37.205387 9111 tablet_copy_client.cc:806] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: tablet copy: Starting download of 0 data blocks...
I20251026 09:11:37.205500 9111 tablet_copy_client.cc:670] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: tablet copy: Starting download of 1 WAL segments...
I20251026 09:11:37.206172 9111 tablet_copy_client.cc:538] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:37.206830 9111 tablet_bootstrap.cc:492] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: Bootstrap starting.
I20251026 09:11:37.208096 9111 tablet_bootstrap.cc:492] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:37.208261 9111 tablet_bootstrap.cc:492] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: Bootstrap complete.
I20251026 09:11:37.208338 9111 ts_tablet_manager.cc:1403] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:37.208446 9111 raft_consensus.cc:359] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:37.208510 9111 raft_consensus.cc:740] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 68b888029fc5492c8246f381b8d634a1, State: Initialized, Role: LEARNER
I20251026 09:11:37.208532 9111 consensus_queue.cc:260] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:37.208618 9111 ts_tablet_manager.cc:1434] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:37.208870 8869 tablet_copy_service.cc:342] P a9b8809cbf5b418eb162e9236454ce45: Request end of tablet copy session 68b888029fc5492c8246f381b8d634a1-95b88dca28844b6a991216145224b07e received from {username='slave'} at 127.0.0.1:37728
I20251026 09:11:37.208935 8869 tablet_copy_service.cc:434] P a9b8809cbf5b418eb162e9236454ce45: ending tablet copy session 68b888029fc5492c8246f381b8d634a1-95b88dca28844b6a991216145224b07e on tablet 95b88dca28844b6a991216145224b07e with peer 68b888029fc5492c8246f381b8d634a1
I20251026 09:11:37.315551 9111 ts_tablet_manager.cc:933] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: Initiating tablet copy from peer a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2:42977)
I20251026 09:11:37.315800 9111 tablet_copy_client.cc:323] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42977
I20251026 09:11:37.316061 8869 tablet_copy_service.cc:140] P a9b8809cbf5b418eb162e9236454ce45: Received BeginTabletCopySession request for tablet c5aeba37843040c5a4bb0422bedd2513 from peer 68b888029fc5492c8246f381b8d634a1 ({username='slave'} at 127.0.0.1:37728)
I20251026 09:11:37.316123 8869 tablet_copy_service.cc:161] P a9b8809cbf5b418eb162e9236454ce45: Beginning new tablet copy session on tablet c5aeba37843040c5a4bb0422bedd2513 from peer 68b888029fc5492c8246f381b8d634a1 at {username='slave'} at 127.0.0.1:37728: session id = 68b888029fc5492c8246f381b8d634a1-c5aeba37843040c5a4bb0422bedd2513
I20251026 09:11:37.316623 8869 tablet_copy_source_session.cc:215] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:11:37.317082 9111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5aeba37843040c5a4bb0422bedd2513. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:37.318163 9111 tablet_copy_client.cc:806] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Starting download of 0 data blocks...
I20251026 09:11:37.318305 9111 tablet_copy_client.cc:670] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Starting download of 1 WAL segments...
I20251026 09:11:37.319139 9111 tablet_copy_client.cc:538] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:37.319909 9111 tablet_bootstrap.cc:492] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: Bootstrap starting.
I20251026 09:11:37.321084 9111 tablet_bootstrap.cc:492] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:37.321251 9111 tablet_bootstrap.cc:492] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: Bootstrap complete.
I20251026 09:11:37.321350 9111 ts_tablet_manager.cc:1403] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:37.321477 9111 raft_consensus.cc:359] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:37.321545 9111 raft_consensus.cc:740] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 68b888029fc5492c8246f381b8d634a1, State: Initialized, Role: LEARNER
I20251026 09:11:37.321581 9111 consensus_queue.cc:260] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:37.321686 9111 ts_tablet_manager.cc:1434] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:37.321935 8869 tablet_copy_service.cc:342] P a9b8809cbf5b418eb162e9236454ce45: Request end of tablet copy session 68b888029fc5492c8246f381b8d634a1-c5aeba37843040c5a4bb0422bedd2513 received from {username='slave'} at 127.0.0.1:37728
I20251026 09:11:37.321982 8869 tablet_copy_service.cc:434] P a9b8809cbf5b418eb162e9236454ce45: ending tablet copy session 68b888029fc5492c8246f381b8d634a1-c5aeba37843040c5a4bb0422bedd2513 on tablet c5aeba37843040c5a4bb0422bedd2513 with peer 68b888029fc5492c8246f381b8d634a1
I20251026 09:11:37.636590 9099 consensus_queue.cc:579] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Leader has been unable to successfully communicate with peer 2cc7a7a32bff4c96b276e4b495ae3fbe for more than 1 seconds (1.052s)
I20251026 09:11:37.669365 9068 raft_consensus.cc:1217] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 09:11:37.773413 9127 consensus_queue.cc:579] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Leader has been unable to successfully communicate with peer 2cc7a7a32bff4c96b276e4b495ae3fbe for more than 1 seconds (1.103s)
I20251026 09:11:37.798208 9068 raft_consensus.cc:1217] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 09:11:37.823155 9068 raft_consensus.cc:1217] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 09:11:37.875520 9131 consensus_queue.cc:579] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Leader has been unable to successfully communicate with peer 2cc7a7a32bff4c96b276e4b495ae3fbe for more than 1 seconds (1.067s)
W20251026 09:11:37.960134 8728 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 2cc7a7a32bff4c96b276e4b495ae3fbe not available for placement
I20251026 09:11:38.259752 9131 raft_consensus.cc:1064] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45: attempting to promote NON_VOTER 68b888029fc5492c8246f381b8d634a1 to VOTER
I20251026 09:11:38.260013 9131 consensus_queue.cc:237] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } }
I20251026 09:11:38.260584 9068 raft_consensus.cc:1275] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:38.260617 8934 raft_consensus.cc:1275] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:38.260919 9131 consensus_queue.cc:1048] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251026 09:11:38.261044 8824 consensus_peers.cc:597] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:38.261068 9133 consensus_queue.cc:1048] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:38.262511 9133 raft_consensus.cc:2955] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.262655 9068 raft_consensus.cc:2955] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.263859 8934 raft_consensus.cc:2955] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.263906 8680 catalog_manager.cc:5649] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 reported cstate change: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:11:38.265363 8859 consensus_queue.cc:237] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } }
I20251026 09:11:38.265751 9068 raft_consensus.cc:1275] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:38.265880 8934 raft_consensus.cc:1275] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:38.265981 9131 consensus_queue.cc:1048] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:38.266104 9131 consensus_queue.cc:1048] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:38.266676 9131 raft_consensus.cc:2955] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.266841 8934 raft_consensus.cc:2955] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.266938 9068 raft_consensus.cc:2955] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.267860 8665 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c5aeba37843040c5a4bb0422bedd2513 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:11:38.267938 8678 catalog_manager.cc:5649] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 reported cstate change: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New cstate: current_term: 1 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251026 09:11:38.269362 8666 catalog_manager.cc:4712] TS 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c5aeba37843040c5a4bb0422bedd2513: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111)
I20251026 09:11:38.270258 9098 consensus_queue.cc:579] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Leader has been unable to successfully communicate with peer 2cc7a7a32bff4c96b276e4b495ae3fbe for more than 1 seconds (1.518s)
I20251026 09:11:38.270927 8859 consensus_queue.cc:237] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:38.271448 8934 raft_consensus.cc:1275] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 09:11:38.271535 8822 consensus_peers.cc:597] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 -> Peer 68b888029fc5492c8246f381b8d634a1 (127.3.14.4:46557): Couldn't send request to peer 68b888029fc5492c8246f381b8d634a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c761acf6dd7d43b08bcb48d9092e1881. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:38.271610 9098 consensus_queue.cc:1048] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 09:11:38.271771 8824 consensus_peers.cc:597] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:38.272270 9133 raft_consensus.cc:2955] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } } }
I20251026 09:11:38.272374 8934 raft_consensus.cc:2955] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } } }
I20251026 09:11:38.273217 8665 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c761acf6dd7d43b08bcb48d9092e1881 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251026 09:11:38.273458 8680 catalog_manager.cc:5649] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 reported cstate change: config changed from index -1 to 2, NON_VOTER 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) added. New cstate: current_term: 2 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } } }
I20251026 09:11:38.280611 9107 raft_consensus.cc:1064] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73: attempting to promote NON_VOTER 68b888029fc5492c8246f381b8d634a1 to VOTER
I20251026 09:11:38.280840 9107 consensus_queue.cc:237] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } }
I20251026 09:11:38.281383 9068 raft_consensus.cc:1275] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Refusing update from remote peer 44797ee52d314ac6a8032d536eb19e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:38.281421 8859 raft_consensus.cc:1275] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Refusing update from remote peer 44797ee52d314ac6a8032d536eb19e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:38.281641 9147 consensus_queue.cc:1048] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:38.281741 9144 consensus_queue.cc:1048] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251026 09:11:38.281832 8899 consensus_peers.cc:597] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:38.282577 9147 raft_consensus.cc:2955] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.282691 8859 raft_consensus.cc:2955] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.283679 9068 raft_consensus.cc:2955] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.283861 8678 catalog_manager.cc:5649] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 reported cstate change: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "44797ee52d314ac6a8032d536eb19e73" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:11:38.284915 8934 consensus_queue.cc:237] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } }
I20251026 09:11:38.285352 9068 raft_consensus.cc:1275] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Refusing update from remote peer 44797ee52d314ac6a8032d536eb19e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:38.285439 8859 raft_consensus.cc:1275] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Refusing update from remote peer 44797ee52d314ac6a8032d536eb19e73: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:38.285512 9144 consensus_queue.cc:1048] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:38.285647 9146 consensus_queue.cc:1048] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:38.286019 9144 raft_consensus.cc:2955] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.286152 9068 raft_consensus.cc:2955] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.286342 8859 raft_consensus.cc:2955] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.287001 8667 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 26e16f5cbf2040f1bdced2c676615707 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:11:38.287317 8678 catalog_manager.cc:5649] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 reported cstate change: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New cstate: current_term: 1 leader_uuid: "44797ee52d314ac6a8032d536eb19e73" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.314504 9131 raft_consensus.cc:1064] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45: attempting to promote NON_VOTER 68b888029fc5492c8246f381b8d634a1 to VOTER
I20251026 09:11:38.314730 9131 consensus_queue.cc:237] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } }
I20251026 09:11:38.315323 9068 raft_consensus.cc:1275] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [term 1 LEARNER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:38.315343 8934 raft_consensus.cc:1275] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:11:38.315575 9099 consensus_queue.cc:1048] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:38.315649 9131 consensus_queue.cc:1048] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251026 09:11:38.315706 8824 consensus_peers.cc:597] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:38.316531 9099 raft_consensus.cc:2955] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.316671 8934 raft_consensus.cc:2955] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.317837 9068 raft_consensus.cc:2955] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.317943 8679 catalog_manager.cc:5649] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 reported cstate change: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:11:38.318962 8859 consensus_queue.cc:237] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } }
I20251026 09:11:38.319425 8934 raft_consensus.cc:1275] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:38.319425 9068 raft_consensus.cc:1275] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:11:38.319628 9133 consensus_queue.cc:1048] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:38.319797 9133 consensus_queue.cc:1048] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:38.320238 9098 raft_consensus.cc:2955] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.320358 8934 raft_consensus.cc:2955] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.321257 8665 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 95b88dca28844b6a991216145224b07e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:11:38.321265 8679 catalog_manager.cc:5649] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 reported cstate change: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New cstate: current_term: 1 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.321614 9068 raft_consensus.cc:2955] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:38.686354 9153 ts_tablet_manager.cc:933] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: Initiating tablet copy from peer a9b8809cbf5b418eb162e9236454ce45 (127.3.14.2:42977)
I20251026 09:11:38.686566 9153 tablet_copy_client.cc:323] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42977
I20251026 09:11:38.686837 8869 tablet_copy_service.cc:140] P a9b8809cbf5b418eb162e9236454ce45: Received BeginTabletCopySession request for tablet c761acf6dd7d43b08bcb48d9092e1881 from peer 68b888029fc5492c8246f381b8d634a1 ({username='slave'} at 127.0.0.1:37728)
I20251026 09:11:38.686911 8869 tablet_copy_service.cc:161] P a9b8809cbf5b418eb162e9236454ce45: Beginning new tablet copy session on tablet c761acf6dd7d43b08bcb48d9092e1881 from peer 68b888029fc5492c8246f381b8d634a1 at {username='slave'} at 127.0.0.1:37728: session id = 68b888029fc5492c8246f381b8d634a1-c761acf6dd7d43b08bcb48d9092e1881
I20251026 09:11:38.687482 8869 tablet_copy_source_session.cc:215] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:11:38.687922 9153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c761acf6dd7d43b08bcb48d9092e1881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:38.689034 9153 tablet_copy_client.cc:806] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Starting download of 0 data blocks...
I20251026 09:11:38.689179 9153 tablet_copy_client.cc:670] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Starting download of 1 WAL segments...
I20251026 09:11:38.689742 9153 tablet_copy_client.cc:538] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:38.690392 9153 tablet_bootstrap.cc:492] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: Bootstrap starting.
I20251026 09:11:38.691699 9153 tablet_bootstrap.cc:492] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:38.691880 9153 tablet_bootstrap.cc:492] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: Bootstrap complete.
I20251026 09:11:38.691972 9153 ts_tablet_manager.cc:1403] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:38.692101 9153 raft_consensus.cc:359] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:38.692164 9153 raft_consensus.cc:740] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 68b888029fc5492c8246f381b8d634a1, State: Initialized, Role: LEARNER
I20251026 09:11:38.692190 9153 consensus_queue.cc:260] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: true } }
I20251026 09:11:38.692276 9153 ts_tablet_manager.cc:1434] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:38.692556 8869 tablet_copy_service.cc:342] P a9b8809cbf5b418eb162e9236454ce45: Request end of tablet copy session 68b888029fc5492c8246f381b8d634a1-c761acf6dd7d43b08bcb48d9092e1881 received from {username='slave'} at 127.0.0.1:37728
I20251026 09:11:38.692631 8869 tablet_copy_service.cc:434] P a9b8809cbf5b418eb162e9236454ce45: ending tablet copy session 68b888029fc5492c8246f381b8d634a1-c761acf6dd7d43b08bcb48d9092e1881 on tablet c761acf6dd7d43b08bcb48d9092e1881 with peer 68b888029fc5492c8246f381b8d634a1
W20251026 09:11:38.960500 8728 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 2cc7a7a32bff4c96b276e4b495ae3fbe not available for placement
I20251026 09:11:39.129983 9068 raft_consensus.cc:1217] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
W20251026 09:11:39.417938 8666 catalog_manager.cc:4712] TS 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26e16f5cbf2040f1bdced2c676615707: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111)
I20251026 09:11:39.653975 9159 raft_consensus.cc:1064] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45: attempting to promote NON_VOTER 68b888029fc5492c8246f381b8d634a1 to VOTER
I20251026 09:11:39.654214 9159 consensus_queue.cc:237] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } }
I20251026 09:11:39.654713 9068 raft_consensus.cc:1275] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [term 2 LEARNER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 09:11:39.654733 8934 raft_consensus.cc:1275] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 09:11:39.654995 9158 consensus_queue.cc:1048] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:39.655085 9157 consensus_queue.cc:1048] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251026 09:11:39.655177 8824 consensus_peers.cc:597] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 -> Peer 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): Couldn't send request to peer 2cc7a7a32bff4c96b276e4b495ae3fbe. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:39.655968 9157 raft_consensus.cc:2955] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:39.656129 8934 raft_consensus.cc:2955] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:39.657339 8680 catalog_manager.cc:5649] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 reported cstate change: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:39.657339 9068 raft_consensus.cc:2955] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 68b888029fc5492c8246f381b8d634a1 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2cc7a7a32bff4c96b276e4b495ae3fbe" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37621 } } peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:39.659075 8859 consensus_queue.cc:237] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } }
I20251026 09:11:39.659439 8934 raft_consensus.cc:1275] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 09:11:39.659439 9068 raft_consensus.cc:1275] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [term 2 FOLLOWER]: Refusing update from remote peer a9b8809cbf5b418eb162e9236454ce45: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 09:11:39.659626 9161 consensus_queue.cc:1048] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:39.659713 9157 consensus_queue.cc:1048] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:11:39.660279 9159 raft_consensus.cc:2955] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:39.660429 9068 raft_consensus.cc:2955] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:39.661412 8665 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c761acf6dd7d43b08bcb48d9092e1881 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:11:39.661538 8934 raft_consensus.cc:2955] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } } }
I20251026 09:11:39.661590 8679 catalog_manager.cc:5649] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 reported cstate change: config changed from index 3 to 4, VOTER 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1) evicted. New cstate: current_term: 2 leader_uuid: "a9b8809cbf5b418eb162e9236454ce45" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9b8809cbf5b418eb162e9236454ce45" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44797ee52d314ac6a8032d536eb19e73" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 41349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68b888029fc5492c8246f381b8d634a1" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251026 09:11:40.445255 8666 catalog_manager.cc:4712] TS 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26e16f5cbf2040f1bdced2c676615707: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111)
W20251026 09:11:41.897343 8666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20251026 09:11:41.898000 8666 catalog_manager.cc:4712] TS 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c761acf6dd7d43b08bcb48d9092e1881: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111)
I20251026 09:11:43.383250 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:11:43.386392 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:43.386530 3128 tablet_replica.cc:333] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45: stopping tablet replica
I20251026 09:11:43.386602 3128 raft_consensus.cc:2243] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:43.386694 3128 raft_consensus.cc:2272] T 26e16f5cbf2040f1bdced2c676615707 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.386921 3128 tablet_replica.cc:333] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45: stopping tablet replica
I20251026 09:11:43.386968 3128 raft_consensus.cc:2243] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:43.387058 3128 raft_consensus.cc:2272] T c5aeba37843040c5a4bb0422bedd2513 P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.388278 3128 tablet_replica.cc:333] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45: stopping tablet replica
I20251026 09:11:43.388345 3128 raft_consensus.cc:2243] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:43.388414 3128 raft_consensus.cc:2272] T 95b88dca28844b6a991216145224b07e P a9b8809cbf5b418eb162e9236454ce45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.388545 3128 tablet_replica.cc:333] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45: stopping tablet replica
I20251026 09:11:43.388579 3128 raft_consensus.cc:2243] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 2 LEADER]: Raft consensus shutting down.
I20251026 09:11:43.388636 3128 raft_consensus.cc:2272] T c761acf6dd7d43b08bcb48d9092e1881 P a9b8809cbf5b418eb162e9236454ce45 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.400566 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:11:43.401957 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:11:43.405050 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:43.405211 3128 tablet_replica.cc:333] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73: stopping tablet replica
I20251026 09:11:43.405301 3128 raft_consensus.cc:2243] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:43.405380 3128 raft_consensus.cc:2272] T 26e16f5cbf2040f1bdced2c676615707 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.405557 3128 tablet_replica.cc:333] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73: stopping tablet replica
I20251026 09:11:43.405592 3128 raft_consensus.cc:2243] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:43.405638 3128 raft_consensus.cc:2272] T c761acf6dd7d43b08bcb48d9092e1881 P 44797ee52d314ac6a8032d536eb19e73 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.405725 3128 tablet_replica.cc:333] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73: stopping tablet replica
I20251026 09:11:43.405755 3128 raft_consensus.cc:2243] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:43.405788 3128 raft_consensus.cc:2272] T 95b88dca28844b6a991216145224b07e P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.405871 3128 tablet_replica.cc:333] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73: stopping tablet replica
I20251026 09:11:43.405901 3128 raft_consensus.cc:2243] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:43.405946 3128 raft_consensus.cc:2272] T c5aeba37843040c5a4bb0422bedd2513 P 44797ee52d314ac6a8032d536eb19e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.417996 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:11:43.419301 3128 tablet_server.cc:178] TabletServer@127.3.14.4:0 shutting down...
I20251026 09:11:43.421691 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:43.421826 3128 tablet_replica.cc:333] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1: stopping tablet replica
I20251026 09:11:43.421881 3128 raft_consensus.cc:2243] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:43.421952 3128 raft_consensus.cc:2272] T c761acf6dd7d43b08bcb48d9092e1881 P 68b888029fc5492c8246f381b8d634a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.422148 3128 tablet_replica.cc:333] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1: stopping tablet replica
I20251026 09:11:43.422194 3128 raft_consensus.cc:2243] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:43.422224 3128 raft_consensus.cc:2272] T 26e16f5cbf2040f1bdced2c676615707 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.422323 3128 tablet_replica.cc:333] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1: stopping tablet replica
I20251026 09:11:43.422358 3128 raft_consensus.cc:2243] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:43.422385 3128 raft_consensus.cc:2272] T c5aeba37843040c5a4bb0422bedd2513 P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.422488 3128 tablet_replica.cc:333] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1: stopping tablet replica
I20251026 09:11:43.422518 3128 raft_consensus.cc:2243] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:43.422559 3128 raft_consensus.cc:2272] T 95b88dca28844b6a991216145224b07e P 68b888029fc5492c8246f381b8d634a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:43.434293 3128 tablet_server.cc:195] TabletServer@127.3.14.4:0 shutdown complete.
I20251026 09:11:43.435539 3128 master.cc:561] Master@127.3.14.62:41575 shutting down...
W20251026 09:11:43.951478 8666 catalog_manager.cc:4712] TS 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c761acf6dd7d43b08bcb48d9092e1881: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111)
W20251026 09:11:46.642378 8666 catalog_manager.cc:4712] TS 2cc7a7a32bff4c96b276e4b495ae3fbe (127.3.14.1:37621): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26e16f5cbf2040f1bdced2c676615707: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37621: connect: Connection refused (error 111)
I20251026 09:11:47.029970 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:47.030128 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:47.030151 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P ddedba7a0c7140b685e8f0ffdc6ed129: stopping tablet replica
I20251026 09:11:47.042461 3128 master.cc:583] Master@127.3.14.62:41575 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13111 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20251026 09:11:47.049829 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:45323
I20251026 09:11:47.050034 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:11:47.051831 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:47.051920 9178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:47.051854 9180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:47.051952 9177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:47.052223 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:47.052260 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:47.052274 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469907052275 us; error 0 us; skew 500 ppm
I20251026 09:11:47.052948 3128 webserver.cc:492] Webserver started at http://127.3.14.62:39025/ using document root <none> and password file <none>
I20251026 09:11:47.053030 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:47.053084 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:47.053134 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:47.053448 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "09c6d9e2dde9462e9b86200c3ffc1a48"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:47.054457 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:47.055002 9185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.055140 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.055186 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "09c6d9e2dde9462e9b86200c3ffc1a48"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:47.055222 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:47.061674 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:47.061902 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:47.064951 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:45323
I20251026 09:11:47.068068 9237 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:45323 every 8 connection(s)
I20251026 09:11:47.068253 9238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:47.069407 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48: Bootstrap starting.
I20251026 09:11:47.069736 9238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:47.070289 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48: No bootstrap required, opened a new log
I20251026 09:11:47.070423 9238 raft_consensus.cc:359] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6d9e2dde9462e9b86200c3ffc1a48" member_type: VOTER }
I20251026 09:11:47.070482 9238 raft_consensus.cc:385] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:47.070497 9238 raft_consensus.cc:740] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c6d9e2dde9462e9b86200c3ffc1a48, State: Initialized, Role: FOLLOWER
I20251026 09:11:47.070551 9238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6d9e2dde9462e9b86200c3ffc1a48" member_type: VOTER }
I20251026 09:11:47.070588 9238 raft_consensus.cc:399] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:11:47.070616 9238 raft_consensus.cc:493] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:11:47.070643 9238 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:47.071188 9238 raft_consensus.cc:515] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6d9e2dde9462e9b86200c3ffc1a48" member_type: VOTER }
I20251026 09:11:47.071249 9238 leader_election.cc:304] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09c6d9e2dde9462e9b86200c3ffc1a48; no voters:
I20251026 09:11:47.071399 9238 leader_election.cc:290] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:11:47.071539 9238 sys_catalog.cc:565] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:11:47.071597 9241 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:47.071771 9241 raft_consensus.cc:697] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 1 LEADER]: Becoming Leader. State: Replica: 09c6d9e2dde9462e9b86200c3ffc1a48, State: Running, Role: LEADER
I20251026 09:11:47.071830 9241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6d9e2dde9462e9b86200c3ffc1a48" member_type: VOTER }
I20251026 09:11:47.072075 9242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09c6d9e2dde9462e9b86200c3ffc1a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6d9e2dde9462e9b86200c3ffc1a48" member_type: VOTER } }
I20251026 09:11:47.072142 9242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:47.072099 9243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09c6d9e2dde9462e9b86200c3ffc1a48. Latest consensus state: current_term: 1 leader_uuid: "09c6d9e2dde9462e9b86200c3ffc1a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c6d9e2dde9462e9b86200c3ffc1a48" member_type: VOTER } }
I20251026 09:11:47.072178 9243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:47.072330 9250 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:11:47.072558 9250 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:11:47.072940 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:11:47.073161 9250 catalog_manager.cc:1357] Generated new cluster ID: ad7d41350390439da4f2b39b7e977cb9
I20251026 09:11:47.073203 9250 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:11:47.086593 9250 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:11:47.087036 9250 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:11:47.096302 9250 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48: Generated new TSK 0
I20251026 09:11:47.096410 9250 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:11:47.104943 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:47.106580 9260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:47.106603 9262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:47.106707 9259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:47.106746 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:47.106968 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:47.107007 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:47.107031 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469907107031 us; error 0 us; skew 500 ppm
I20251026 09:11:47.107632 3128 webserver.cc:492] Webserver started at http://127.3.14.1:32899/ using document root <none> and password file <none>
I20251026 09:11:47.107728 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:47.107787 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:47.107828 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:47.108098 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "244142e66bf8451f9c8d3c584c6c8c8d"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:47.109078 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:47.109525 9267 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.109663 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.109714 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "244142e66bf8451f9c8d3c584c6c8c8d"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:47.109757 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:47.118393 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:47.118647 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:47.118791 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:47.119014 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:47.119045 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.119071 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:47.119099 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.122097 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:37213
I20251026 09:11:47.122133 9330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:37213 every 8 connection(s)
I20251026 09:11:47.122541 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:47.123874 9336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:47.123942 9338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:47.123901 9335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:47.124249 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:47.124377 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:47.124413 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:47.124432 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469907124433 us; error 0 us; skew 500 ppm
I20251026 09:11:47.125303 3128 webserver.cc:492] Webserver started at http://127.3.14.2:34547/ using document root <none> and password file <none>
I20251026 09:11:47.125412 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:47.125479 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:47.125545 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:47.125888 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "d49cd53f5dd64a25a511de611a4906ff"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:47.126269 9331 heartbeater.cc:344] Connected to a master server at 127.3.14.62:45323
I20251026 09:11:47.126333 9331 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:47.126446 9331 heartbeater.cc:507] Master 127.3.14.62:45323 requested a full tablet report, sending...
I20251026 09:11:47.126689 9202 ts_manager.cc:194] Registered new tserver with Master: 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213)
I20251026 09:11:47.127061 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:47.127431 9202 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57126
I20251026 09:11:47.127578 9343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.127718 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.127768 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "d49cd53f5dd64a25a511de611a4906ff"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:47.127823 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:47.134717 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:47.134994 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:47.135167 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:47.135430 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:47.135462 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.135488 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:47.135520 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.138327 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:39035
I20251026 09:11:47.138392 9406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:39035 every 8 connection(s)
I20251026 09:11:47.138762 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:47.140097 9410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:47.140120 9411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:47.140355 9413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:47.140817 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:47.140926 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:47.140961 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:47.140978 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469907140978 us; error 0 us; skew 500 ppm
I20251026 09:11:47.141784 3128 webserver.cc:492] Webserver started at http://127.3.14.3:44561/ using document root <none> and password file <none>
I20251026 09:11:47.141896 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:47.141953 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:47.142001 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:47.142338 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "226d829f3750486bbe2e8ca2667b36df"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:47.142379 9407 heartbeater.cc:344] Connected to a master server at 127.3.14.62:45323
I20251026 09:11:47.142438 9407 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:47.142556 9407 heartbeater.cc:507] Master 127.3.14.62:45323 requested a full tablet report, sending...
I20251026 09:11:47.142812 9202 ts_manager.cc:194] Registered new tserver with Master: d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035)
I20251026 09:11:47.143158 9202 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57132
I20251026 09:11:47.143445 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:47.143966 9418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.144099 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.144145 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "226d829f3750486bbe2e8ca2667b36df"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:47.144203 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:47.152667 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:47.152979 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:47.153136 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:47.153348 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:47.153379 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.153415 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:47.153442 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:47.156836 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:34513
I20251026 09:11:47.156914 9481 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:34513 every 8 connection(s)
I20251026 09:11:47.160485 9482 heartbeater.cc:344] Connected to a master server at 127.3.14.62:45323
I20251026 09:11:47.160562 9482 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:47.160656 9482 heartbeater.cc:507] Master 127.3.14.62:45323 requested a full tablet report, sending...
I20251026 09:11:47.160905 9202 ts_manager.cc:194] Registered new tserver with Master: 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:47.161260 9202 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57134
I20251026 09:11:47.161482 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.0044502s
I20251026 09:11:48.128057 9331 heartbeater.cc:499] Master 127.3.14.62:45323 was elected leader, sending a full tablet report...
I20251026 09:11:48.143715 9407 heartbeater.cc:499] Master 127.3.14.62:45323 was elected leader, sending a full tablet report...
I20251026 09:11:48.161839 9482 heartbeater.cc:499] Master 127.3.14.62:45323 was elected leader, sending a full tablet report...
I20251026 09:11:48.187006 3128 test_util.cc:276] Using random seed: 740882298
I20251026 09:11:48.192847 9202 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57138:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:11:48.193060 9202 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:11:48.199229 9295 tablet_service.cc:1505] Processing CreateTablet for tablet cbef2b4cd5ab404e89c48a8452f2fe1c (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:11:48.199318 9371 tablet_service.cc:1505] Processing CreateTablet for tablet cbef2b4cd5ab404e89c48a8452f2fe1c (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:11:48.199416 9368 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8f80e14b9a4606b0ee24503fb879c9 (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:11:48.199465 9295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbef2b4cd5ab404e89c48a8452f2fe1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.199499 9368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8f80e14b9a4606b0ee24503fb879c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.199385 9369 tablet_service.cc:1505] Processing CreateTablet for tablet e97341eb7bf94ce6901ad16486ba4138 (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:11:48.199561 9446 tablet_service.cc:1505] Processing CreateTablet for tablet cbef2b4cd5ab404e89c48a8452f2fe1c (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:11:48.199596 9369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e97341eb7bf94ce6901ad16486ba4138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.199653 9446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbef2b4cd5ab404e89c48a8452f2fe1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.199857 9445 tablet_service.cc:1505] Processing CreateTablet for tablet 1a0d9035730140a4b624f225743a2ae5 (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:11:48.199887 9294 tablet_service.cc:1505] Processing CreateTablet for tablet 1a0d9035730140a4b624f225743a2ae5 (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:11:48.199959 9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0d9035730140a4b624f225743a2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.200018 9293 tablet_service.cc:1505] Processing CreateTablet for tablet e97341eb7bf94ce6901ad16486ba4138 (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:11:48.200153 9292 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8f80e14b9a4606b0ee24503fb879c9 (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:11:48.199965 9294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0d9035730140a4b624f225743a2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.199321 9370 tablet_service.cc:1505] Processing CreateTablet for tablet 1a0d9035730140a4b624f225743a2ae5 (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:11:48.201148 9370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0d9035730140a4b624f225743a2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.201570 9444 tablet_service.cc:1505] Processing CreateTablet for tablet e97341eb7bf94ce6901ad16486ba4138 (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:11:48.201658 9444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e97341eb7bf94ce6901ad16486ba4138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.202154 9293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e97341eb7bf94ce6901ad16486ba4138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.202455 9502 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap starting.
I20251026 09:11:48.202560 9371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbef2b4cd5ab404e89c48a8452f2fe1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.202929 9502 tablet_bootstrap.cc:654] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.203011 9443 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8f80e14b9a4606b0ee24503fb879c9 (DEFAULT_TABLE table=test-workload [id=3ad6cc794aed4907b9caee76eaf79315]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:11:48.203101 9443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8f80e14b9a4606b0ee24503fb879c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.203446 9292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8f80e14b9a4606b0ee24503fb879c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:48.203586 9502 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: No bootstrap required, opened a new log
I20251026 09:11:48.203630 9502 ts_tablet_manager.cc:1403] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:48.203814 9502 raft_consensus.cc:359] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.203882 9502 raft_consensus.cc:385] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.203909 9502 raft_consensus.cc:740] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.203966 9502 consensus_queue.cc:260] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.204128 9502 ts_tablet_manager.cc:1434] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:48.204197 9502 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap starting.
I20251026 09:11:48.204612 9502 tablet_bootstrap.cc:654] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.205302 9502 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: No bootstrap required, opened a new log
I20251026 09:11:48.205361 9502 ts_tablet_manager.cc:1403] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:48.205514 9502 raft_consensus.cc:359] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.205596 9502 raft_consensus.cc:385] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.205607 9503 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap starting.
I20251026 09:11:48.205622 9502 raft_consensus.cc:740] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.205693 9502 consensus_queue.cc:260] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.205883 9502 ts_tablet_manager.cc:1434] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:48.205953 9502 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap starting.
I20251026 09:11:48.206089 9503 tablet_bootstrap.cc:654] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.206350 9502 tablet_bootstrap.cc:654] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.206456 9504 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap starting.
I20251026 09:11:48.206924 9504 tablet_bootstrap.cc:654] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.206962 9502 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: No bootstrap required, opened a new log
I20251026 09:11:48.206998 9502 ts_tablet_manager.cc:1403] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:48.207185 9502 raft_consensus.cc:359] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.207268 9502 raft_consensus.cc:385] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.207293 9502 raft_consensus.cc:740] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.207362 9502 consensus_queue.cc:260] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.207466 9502 ts_tablet_manager.cc:1434] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:48.207535 9502 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: Bootstrap starting.
I20251026 09:11:48.207568 9504 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: No bootstrap required, opened a new log
I20251026 09:11:48.207600 9504 ts_tablet_manager.cc:1403] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:48.207749 9504 raft_consensus.cc:359] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.207808 9504 raft_consensus.cc:385] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.207826 9504 raft_consensus.cc:740] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.207942 9502 tablet_bootstrap.cc:654] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.208031 9503 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: No bootstrap required, opened a new log
I20251026 09:11:48.208076 9503 ts_tablet_manager.cc:1403] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:48.208222 9503 raft_consensus.cc:359] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.208287 9503 raft_consensus.cc:385] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.208310 9503 raft_consensus.cc:740] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.208384 9503 consensus_queue.cc:260] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.208520 9502 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: No bootstrap required, opened a new log
I20251026 09:11:48.208572 9503 ts_tablet_manager.cc:1434] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:48.208693 9503 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap starting.
I20251026 09:11:48.208573 9502 ts_tablet_manager.cc:1403] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:48.208938 9502 raft_consensus.cc:359] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:48.208998 9502 raft_consensus.cc:385] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.209021 9502 raft_consensus.cc:740] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.209074 9502 consensus_queue.cc:260] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:48.209168 9502 ts_tablet_manager.cc:1434] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:48.209170 9503 tablet_bootstrap.cc:654] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.209198 9504 consensus_queue.cc:260] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.209419 9504 ts_tablet_manager.cc:1434] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:48.209583 9504 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap starting.
I20251026 09:11:48.210115 9504 tablet_bootstrap.cc:654] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.210733 9504 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: No bootstrap required, opened a new log
I20251026 09:11:48.210733 9503 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: No bootstrap required, opened a new log
I20251026 09:11:48.210806 9504 ts_tablet_manager.cc:1403] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:48.210816 9503 ts_tablet_manager.cc:1403] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:48.211004 9504 raft_consensus.cc:359] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:48.210991 9503 raft_consensus.cc:359] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.211105 9504 raft_consensus.cc:385] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.211119 9503 raft_consensus.cc:385] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.211133 9504 raft_consensus.cc:740] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.211174 9503 raft_consensus.cc:740] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.211207 9504 consensus_queue.cc:260] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:48.211228 9503 consensus_queue.cc:260] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.211304 9504 ts_tablet_manager.cc:1434] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:48.211319 9503 ts_tablet_manager.cc:1434] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:48.211443 9503 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap starting.
I20251026 09:11:48.211520 9504 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap starting.
I20251026 09:11:48.211928 9503 tablet_bootstrap.cc:654] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.211988 9504 tablet_bootstrap.cc:654] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.212599 9503 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: No bootstrap required, opened a new log
I20251026 09:11:48.212599 9504 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: No bootstrap required, opened a new log
I20251026 09:11:48.212693 9503 ts_tablet_manager.cc:1403] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:48.212693 9504 ts_tablet_manager.cc:1403] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:48.212900 9504 raft_consensus.cc:359] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.212903 9503 raft_consensus.cc:359] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.213001 9504 raft_consensus.cc:385] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.213011 9503 raft_consensus.cc:385] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.213047 9504 raft_consensus.cc:740] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.213050 9503 raft_consensus.cc:740] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.213116 9503 consensus_queue.cc:260] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.213120 9504 consensus_queue.cc:260] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.213240 9503 ts_tablet_manager.cc:1434] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:48.213243 9504 ts_tablet_manager.cc:1434] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:48.213325 9504 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap starting.
I20251026 09:11:48.213784 9504 tablet_bootstrap.cc:654] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.213325 9503 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: Bootstrap starting.
I20251026 09:11:48.214425 9504 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: No bootstrap required, opened a new log
I20251026 09:11:48.214478 9504 ts_tablet_manager.cc:1403] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:48.214641 9503 tablet_bootstrap.cc:654] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:48.214618 9504 raft_consensus.cc:359] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.214669 9504 raft_consensus.cc:385] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.214684 9504 raft_consensus.cc:740] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.214727 9504 consensus_queue.cc:260] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.214799 9504 ts_tablet_manager.cc:1434] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:48.215358 9503 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: No bootstrap required, opened a new log
I20251026 09:11:48.215443 9503 ts_tablet_manager.cc:1403] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:11:48.215621 9503 raft_consensus.cc:359] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:48.215682 9503 raft_consensus.cc:385] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:48.215695 9503 raft_consensus.cc:740] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Initialized, Role: FOLLOWER
I20251026 09:11:48.215736 9503 consensus_queue.cc:260] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:48.215826 9503 ts_tablet_manager.cc:1434] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:48.347160 9514 raft_consensus.cc:493] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:48.347270 9514 raft_consensus.cc:515] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.347541 9514 leader_election.cc:290] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:48.350401 9456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df" is_pre_election: true
I20251026 09:11:48.350401 9381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff" is_pre_election: true
I20251026 09:11:48.350528 9456 raft_consensus.cc:2468] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 0.
I20251026 09:11:48.350551 9381 raft_consensus.cc:2468] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 0.
I20251026 09:11:48.350692 9270 leader_election.cc:304] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, 244142e66bf8451f9c8d3c584c6c8c8d; no voters:
I20251026 09:11:48.350831 9514 raft_consensus.cc:2804] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:48.350874 9514 raft_consensus.cc:493] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:48.350903 9514 raft_consensus.cc:3060] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.351677 9514 raft_consensus.cc:515] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.351857 9514 leader_election.cc:290] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 election: Requested vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:48.352031 9456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df"
I20251026 09:11:48.352046 9381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff"
I20251026 09:11:48.352111 9456 raft_consensus.cc:3060] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.352128 9381 raft_consensus.cc:3060] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.352754 9456 raft_consensus.cc:2468] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 1.
I20251026 09:11:48.352844 9381 raft_consensus.cc:2468] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 1.
I20251026 09:11:48.352952 9270 leader_election.cc:304] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, 244142e66bf8451f9c8d3c584c6c8c8d; no voters:
I20251026 09:11:48.353044 9514 raft_consensus.cc:2804] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:48.353209 9514 raft_consensus.cc:697] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 LEADER]: Becoming Leader. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Running, Role: LEADER
I20251026 09:11:48.353271 9514 consensus_queue.cc:237] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.353919 9201 catalog_manager.cc:5649] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1). New cstate: current_term: 1 leader_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: UNKNOWN } } }
W20251026 09:11:48.373451 9332 tablet.cc:2378] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 09:11:48.408901 9514 raft_consensus.cc:493] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:48.409035 9514 raft_consensus.cc:515] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:48.409230 9514 leader_election.cc:290] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:48.409478 9381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbef2b4cd5ab404e89c48a8452f2fe1c" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff" is_pre_election: true
I20251026 09:11:48.409478 9456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbef2b4cd5ab404e89c48a8452f2fe1c" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df" is_pre_election: true
I20251026 09:11:48.409593 9381 raft_consensus.cc:2468] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 0.
I20251026 09:11:48.409595 9456 raft_consensus.cc:2468] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 0.
I20251026 09:11:48.409785 9269 leader_election.cc:304] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244142e66bf8451f9c8d3c584c6c8c8d, d49cd53f5dd64a25a511de611a4906ff; no voters:
I20251026 09:11:48.409885 9514 raft_consensus.cc:2804] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:48.409924 9514 raft_consensus.cc:493] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:48.409955 9514 raft_consensus.cc:3060] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.412452 9514 raft_consensus.cc:515] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:48.412626 9514 leader_election.cc:290] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 election: Requested vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:48.412972 9456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbef2b4cd5ab404e89c48a8452f2fe1c" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df"
I20251026 09:11:48.412990 9381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbef2b4cd5ab404e89c48a8452f2fe1c" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff"
I20251026 09:11:48.413098 9381 raft_consensus.cc:3060] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.413097 9456 raft_consensus.cc:3060] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.413885 9456 raft_consensus.cc:2468] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 1.
I20251026 09:11:48.413885 9381 raft_consensus.cc:2468] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 1.
I20251026 09:11:48.414168 9269 leader_election.cc:304] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244142e66bf8451f9c8d3c584c6c8c8d, d49cd53f5dd64a25a511de611a4906ff; no voters:
I20251026 09:11:48.414291 9514 raft_consensus.cc:2804] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:48.414363 9514 raft_consensus.cc:697] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 LEADER]: Becoming Leader. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Running, Role: LEADER
I20251026 09:11:48.414430 9514 consensus_queue.cc:237] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:48.415096 9201 catalog_manager.cc:5649] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1). New cstate: current_term: 1 leader_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } health_report { overall_health: HEALTHY } } }
I20251026 09:11:48.444381 9514 raft_consensus.cc:493] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:48.444504 9514 raft_consensus.cc:515] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.444689 9514 leader_election.cc:290] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:48.444947 9456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8f80e14b9a4606b0ee24503fb879c9" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df" is_pre_election: true
I20251026 09:11:48.445037 9456 raft_consensus.cc:2468] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 0.
I20251026 09:11:48.445019 9381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8f80e14b9a4606b0ee24503fb879c9" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff" is_pre_election: true
I20251026 09:11:48.445165 9381 raft_consensus.cc:2468] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 0.
I20251026 09:11:48.445258 9270 leader_election.cc:304] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, 244142e66bf8451f9c8d3c584c6c8c8d; no voters:
I20251026 09:11:48.445367 9514 raft_consensus.cc:2804] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:48.445427 9514 raft_consensus.cc:493] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:48.445443 9514 raft_consensus.cc:3060] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.446156 9514 raft_consensus.cc:515] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.446305 9514 leader_election.cc:290] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 election: Requested vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:48.446497 9381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8f80e14b9a4606b0ee24503fb879c9" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff"
I20251026 09:11:48.446501 9456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8f80e14b9a4606b0ee24503fb879c9" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df"
I20251026 09:11:48.446563 9381 raft_consensus.cc:3060] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.446583 9456 raft_consensus.cc:3060] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.447173 9456 raft_consensus.cc:2468] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 1.
I20251026 09:11:48.447180 9381 raft_consensus.cc:2468] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 1.
I20251026 09:11:48.447384 9270 leader_election.cc:304] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, 244142e66bf8451f9c8d3c584c6c8c8d; no voters:
I20251026 09:11:48.447499 9514 raft_consensus.cc:2804] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:48.447558 9514 raft_consensus.cc:697] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 LEADER]: Becoming Leader. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Running, Role: LEADER
I20251026 09:11:48.447603 9514 consensus_queue.cc:237] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.448310 9201 catalog_manager.cc:5649] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1). New cstate: current_term: 1 leader_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:48.545147 9506 raft_consensus.cc:493] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:48.545270 9506 raft_consensus.cc:515] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.545478 9506 leader_election.cc:290] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:48.548484 9456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e97341eb7bf94ce6901ad16486ba4138" candidate_uuid: "d49cd53f5dd64a25a511de611a4906ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df" is_pre_election: true
I20251026 09:11:48.548530 9305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e97341eb7bf94ce6901ad16486ba4138" candidate_uuid: "d49cd53f5dd64a25a511de611a4906ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" is_pre_election: true
I20251026 09:11:48.548625 9456 raft_consensus.cc:2468] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49cd53f5dd64a25a511de611a4906ff in term 0.
I20251026 09:11:48.548629 9305 raft_consensus.cc:2468] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49cd53f5dd64a25a511de611a4906ff in term 0.
I20251026 09:11:48.548835 9346 leader_election.cc:304] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244142e66bf8451f9c8d3c584c6c8c8d, d49cd53f5dd64a25a511de611a4906ff; no voters:
I20251026 09:11:48.548959 9506 raft_consensus.cc:2804] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:11:48.549008 9506 raft_consensus.cc:493] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:48.549028 9506 raft_consensus.cc:3060] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.549685 9506 raft_consensus.cc:515] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.549816 9506 leader_election.cc:290] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [CANDIDATE]: Term 1 election: Requested vote from peers 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:48.550019 9305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e97341eb7bf94ce6901ad16486ba4138" candidate_uuid: "d49cd53f5dd64a25a511de611a4906ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244142e66bf8451f9c8d3c584c6c8c8d"
I20251026 09:11:48.550082 9305 raft_consensus.cc:3060] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.550040 9456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e97341eb7bf94ce6901ad16486ba4138" candidate_uuid: "d49cd53f5dd64a25a511de611a4906ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df"
I20251026 09:11:48.550112 9456 raft_consensus.cc:3060] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:48.550683 9305 raft_consensus.cc:2468] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d49cd53f5dd64a25a511de611a4906ff in term 1.
I20251026 09:11:48.550710 9456 raft_consensus.cc:2468] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d49cd53f5dd64a25a511de611a4906ff in term 1.
I20251026 09:11:48.550861 9346 leader_election.cc:304] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244142e66bf8451f9c8d3c584c6c8c8d, d49cd53f5dd64a25a511de611a4906ff; no voters:
I20251026 09:11:48.550961 9506 raft_consensus.cc:2804] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:48.551021 9506 raft_consensus.cc:697] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 LEADER]: Becoming Leader. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Running, Role: LEADER
I20251026 09:11:48.551079 9506 consensus_queue.cc:237] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:48.551774 9201 catalog_manager.cc:5649] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff reported cstate change: term changed from 0 to 1, leader changed from <none> to d49cd53f5dd64a25a511de611a4906ff (127.3.14.2). New cstate: current_term: 1 leader_uuid: "d49cd53f5dd64a25a511de611a4906ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:48.553469 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:11:48.556093 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:48.556303 3128 tablet_replica.cc:333] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: stopping tablet replica
I20251026 09:11:48.556391 3128 raft_consensus.cc:2243] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:48.556447 3128 raft_consensus.cc:2272] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.556751 3128 tablet_replica.cc:333] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: stopping tablet replica
I20251026 09:11:48.556802 3128 raft_consensus.cc:2243] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:48.556856 3128 pending_rounds.cc:70] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Trying to abort 1 pending ops.
I20251026 09:11:48.556885 3128 pending_rounds.cc:77] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214980744615583744 op_type: NO_OP noop_request { }
I20251026 09:11:48.556943 3128 raft_consensus.cc:2889] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 09:11:48.556963 3128 raft_consensus.cc:2272] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.557251 3128 tablet_replica.cc:333] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: stopping tablet replica
I20251026 09:11:48.557303 3128 raft_consensus.cc:2243] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:48.557348 3128 pending_rounds.cc:70] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Trying to abort 1 pending ops.
I20251026 09:11:48.557372 3128 pending_rounds.cc:77] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214980745001906176 op_type: NO_OP noop_request { }
I20251026 09:11:48.557417 3128 raft_consensus.cc:2889] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 09:11:48.557447 3128 raft_consensus.cc:2272] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.557677 3128 tablet_replica.cc:333] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: stopping tablet replica
I20251026 09:11:48.557720 3128 raft_consensus.cc:2243] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:48.557767 3128 pending_rounds.cc:70] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Trying to abort 1 pending ops.
I20251026 09:11:48.557794 3128 pending_rounds.cc:77] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214980744866152448 op_type: NO_OP noop_request { }
I20251026 09:11:48.557833 3128 raft_consensus.cc:2889] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 09:11:48.557863 3128 raft_consensus.cc:2272] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.570122 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:11:48.571416 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:11:48.574882 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:48.575064 3128 tablet_replica.cc:333] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: stopping tablet replica
I20251026 09:11:48.575182 3128 raft_consensus.cc:2243] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:48.575253 3128 pending_rounds.cc:70] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Trying to abort 1 pending ops.
I20251026 09:11:48.575271 3128 pending_rounds.cc:77] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214980745425993728 op_type: NO_OP noop_request { }
I20251026 09:11:48.575352 3128 raft_consensus.cc:2889] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 09:11:48.575394 3128 raft_consensus.cc:2272] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.575696 3128 tablet_replica.cc:333] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: stopping tablet replica
I20251026 09:11:48.575750 3128 raft_consensus.cc:2243] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:48.575778 3128 raft_consensus.cc:2272] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.575973 3128 tablet_replica.cc:333] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: stopping tablet replica
I20251026 09:11:48.576014 3128 raft_consensus.cc:2243] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:48.576043 3128 raft_consensus.cc:2272] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.576211 3128 tablet_replica.cc:333] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: stopping tablet replica
I20251026 09:11:48.576248 3128 raft_consensus.cc:2243] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:48.576273 3128 raft_consensus.cc:2272] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.588271 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:11:48.589478 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:11:48.592352 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:48.592550 3128 tablet_replica.cc:333] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: stopping tablet replica
I20251026 09:11:48.592612 3128 raft_consensus.cc:2243] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:48.592680 3128 raft_consensus.cc:2272] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.592919 3128 tablet_replica.cc:333] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: stopping tablet replica
I20251026 09:11:48.592967 3128 raft_consensus.cc:2243] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:48.593003 3128 raft_consensus.cc:2272] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.593219 3128 tablet_replica.cc:333] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: stopping tablet replica
I20251026 09:11:48.593258 3128 raft_consensus.cc:2243] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:48.593281 3128 raft_consensus.cc:2272] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.593415 3128 tablet_replica.cc:333] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: stopping tablet replica
I20251026 09:11:48.593446 3128 raft_consensus.cc:2243] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:48.593470 3128 raft_consensus.cc:2272] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:48.605310 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:11:48.606664 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:48.608060 9533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:48.608042 9530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:48.608172 9531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:48.608219 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:48.608451 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:48.608505 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:48.608536 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469908608537 us; error 0 us; skew 500 ppm
I20251026 09:11:48.609189 3128 webserver.cc:492] Webserver started at http://127.3.14.4:38773/ using document root <none> and password file <none>
I20251026 09:11:48.609292 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:48.609351 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:48.609409 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:48.609706 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-3-root/instance:
uuid: "867eac79b8c74013b9e5cad031815cca"
format_stamp: "Formatted at 2025-10-26 09:11:48 on dist-test-slave-w8k0"
I20251026 09:11:48.610664 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:48.611217 9538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:48.611407 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:48.611470 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-3-root
uuid: "867eac79b8c74013b9e5cad031815cca"
format_stamp: "Formatted at 2025-10-26 09:11:48 on dist-test-slave-w8k0"
I20251026 09:11:48.611519 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:48.621831 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:48.622138 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:48.622355 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:48.622597 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:48.622629 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:48.622660 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:48.622681 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:48.625639 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.4:34947
I20251026 09:11:48.627880 9601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.4:34947 every 8 connection(s)
I20251026 09:11:48.631168 9602 heartbeater.cc:344] Connected to a master server at 127.3.14.62:45323
I20251026 09:11:48.631242 9602 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:48.631378 9602 heartbeater.cc:507] Master 127.3.14.62:45323 requested a full tablet report, sending...
I20251026 09:11:48.631652 9201 ts_manager.cc:194] Registered new tserver with Master: 867eac79b8c74013b9e5cad031815cca (127.3.14.4:34947)
I20251026 09:11:48.632025 9201 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57150
W20251026 09:11:49.072937 9249 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table 3ad6cc794aed4907b9caee76eaf79315: could not find any suitable replica to move from server 244142e66bf8451f9c8d3c584c6c8c8d to server 867eac79b8c74013b9e5cad031815cca
I20251026 09:11:49.632586 9602 heartbeater.cc:499] Master 127.3.14.62:45323 was elected leader, sending a full tablet report...
W20251026 09:11:50.074080 9249 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:50.074571 9187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20251026 09:11:50.074958 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:50.075009 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:50.075706 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:50.075758 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:50.076176 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:50.076222 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:51.077359 9249 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:51.078265 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:51.078332 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:51.078922 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:51.078958 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:51.079412 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:51.079450 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:52.080569 9249 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:52.081414 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:52.081466 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:52.081982 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:52.082026 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:52.082521 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:52.082573 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:53.083844 9249 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:53.084704 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:53.084764 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.2:39035: connect: Connection refused (error 111)
W20251026 09:11:53.085366 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:53.085431 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:53.086017 9249 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:53.086069 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.3.14.1:37213: connect: Connection refused (error 111)
W20251026 09:11:54.086354 9249 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver d49cd53f5dd64a25a511de611a4906ff not available for placement
W20251026 09:11:55.086745 9249 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver d49cd53f5dd64a25a511de611a4906ff not available for placement
I20251026 09:11:55.652787 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:55.654816 9617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:55.654816 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:55.654933 9620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:55.655047 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:55.655200 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:55.655241 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:55.655256 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469915655255 us; error 0 us; skew 500 ppm
I20251026 09:11:55.655994 3128 webserver.cc:492] Webserver started at http://127.3.14.1:32899/ using document root <none> and password file <none>
I20251026 09:11:55.656127 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:55.656169 3128 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 09:11:55.656724 3128 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:55.657379 9625 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:55.657588 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:55.657655 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "244142e66bf8451f9c8d3c584c6c8c8d"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:55.657727 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:55.668412 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:55.668681 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:55.668856 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:55.669239 9632 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
I20251026 09:11:55.670668 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20251026 09:11:55.670747 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20251026 09:11:55.670799 3128 ts_tablet_manager.cc:600] Registering tablets (0/4 complete)
I20251026 09:11:55.671375 9632 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap starting.
I20251026 09:11:55.671996 3128 ts_tablet_manager.cc:616] Registered 4 tablets
I20251026 09:11:55.672039 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:55.673040 9632 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 09:11:55.673326 9632 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap complete.
I20251026 09:11:55.673440 9632 ts_tablet_manager.cc:1403] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251026 09:11:55.673604 9632 raft_consensus.cc:359] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.673696 9632 raft_consensus.cc:740] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.673799 9632 consensus_queue.cc:260] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.673969 9632 ts_tablet_manager.cc:1434] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:55.674052 9632 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap starting.
I20251026 09:11:55.675082 9632 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 09:11:55.675311 9632 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap complete.
I20251026 09:11:55.675465 9632 ts_tablet_manager.cc:1403] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:55.675601 9632 raft_consensus.cc:359] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.675691 9632 raft_consensus.cc:740] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.675755 9632 consensus_queue.cc:260] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.675853 9632 ts_tablet_manager.cc:1434] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:55.675894 9632 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap starting.
I20251026 09:11:55.676112 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:37213
I20251026 09:11:55.676177 9692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:37213 every 8 connection(s)
I20251026 09:11:55.676412 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:11:55.676952 9632 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:55.677161 9632 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap complete.
I20251026 09:11:55.677282 9632 ts_tablet_manager.cc:1403] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:55.677413 9632 raft_consensus.cc:359] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.677490 9632 raft_consensus.cc:740] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.677549 9632 consensus_queue.cc:260] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.677716 9632 ts_tablet_manager.cc:1434] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20251026 09:11:55.677719 9698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:55.677793 9632 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap starting.
W20251026 09:11:55.678022 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:55.677982 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:55.678804 9632 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 09:11:55.679016 9632 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Bootstrap complete.
I20251026 09:11:55.679113 9632 ts_tablet_manager.cc:1403] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:55.679252 9632 raft_consensus.cc:359] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:55.679361 9632 raft_consensus.cc:740] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.679426 9632 consensus_queue.cc:260] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:55.679546 9632 ts_tablet_manager.cc:1434] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20251026 09:11:55.679965 9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:55.680114 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:55.680167 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:55.680187 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469915680187 us; error 0 us; skew 500 ppm
I20251026 09:11:55.680318 9693 heartbeater.cc:344] Connected to a master server at 127.3.14.62:45323
I20251026 09:11:55.680385 9693 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:55.680528 9693 heartbeater.cc:507] Master 127.3.14.62:45323 requested a full tablet report, sending...
I20251026 09:11:55.680931 9201 ts_manager.cc:194] Re-registered known tserver with Master: 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213)
I20251026 09:11:55.680951 3128 webserver.cc:492] Webserver started at http://127.3.14.2:34547/ using document root <none> and password file <none>
I20251026 09:11:55.681066 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:55.681105 3128 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 09:11:55.681533 9201 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36394
I20251026 09:11:55.681967 9693 heartbeater.cc:499] Master 127.3.14.62:45323 was elected leader, sending a full tablet report...
I20251026 09:11:55.682156 3128 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:55.682950 9707 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:55.683097 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:55.683161 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "d49cd53f5dd64a25a511de611a4906ff"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:55.683220 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:55.688336 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:55.688613 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:55.688794 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:55.690325 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20251026 09:11:55.690394 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 09:11:55.690915 9714 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap starting.
I20251026 09:11:55.691535 3128 ts_tablet_manager.cc:616] Registered 4 tablets
I20251026 09:11:55.691582 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:55.692126 9714 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:55.692309 9714 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap complete.
I20251026 09:11:55.692420 9714 ts_tablet_manager.cc:1403] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:11:55.692623 9714 raft_consensus.cc:359] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.692754 9714 raft_consensus.cc:740] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.692843 9714 consensus_queue.cc:260] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.693038 9714 ts_tablet_manager.cc:1434] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:55.693109 9714 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap starting.
I20251026 09:11:55.694437 9714 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:55.694643 9714 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap complete.
I20251026 09:11:55.694768 9714 ts_tablet_manager.cc:1403] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251026 09:11:55.694957 9714 raft_consensus.cc:359] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.695022 9714 raft_consensus.cc:740] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.695079 9714 consensus_queue.cc:260] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.695181 9714 ts_tablet_manager.cc:1434] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:55.695248 9714 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap starting.
I20251026 09:11:55.695472 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:39035
I20251026 09:11:55.695560 9774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:39035 every 8 connection(s)
I20251026 09:11:55.695914 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:11:55.696916 9714 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 09:11:55.697155 9714 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Bootstrap complete.
I20251026 09:11:55.697275 9714 ts_tablet_manager.cc:1403] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
W20251026 09:11:55.697417 9781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:55.697503 9714 raft_consensus.cc:359] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.697595 9714 raft_consensus.cc:740] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.697666 9714 consensus_queue.cc:260] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.697813 9714 ts_tablet_manager.cc:1434] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20251026 09:11:55.697830 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:55.697877 9714 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: Bootstrap starting.
I20251026 09:11:55.698132 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:55.698273 9779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:55.698462 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:55.698498 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:55.698518 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469915698518 us; error 0 us; skew 500 ppm
I20251026 09:11:55.699295 3128 webserver.cc:492] Webserver started at http://127.3.14.3:44561/ using document root <none> and password file <none>
I20251026 09:11:55.699415 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:55.699447 3128 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 09:11:55.699968 9775 heartbeater.cc:344] Connected to a master server at 127.3.14.62:45323
I20251026 09:11:55.700045 9775 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:55.700065 3128 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:55.699990 9714 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:55.700217 9775 heartbeater.cc:507] Master 127.3.14.62:45323 requested a full tablet report, sending...
I20251026 09:11:55.700291 9714 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: Bootstrap complete.
I20251026 09:11:55.700515 9201 ts_manager.cc:194] Re-registered known tserver with Master: d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035)
I20251026 09:11:55.700531 9714 ts_tablet_manager.cc:1403] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251026 09:11:55.700718 9714 raft_consensus.cc:359] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:55.700812 9714 raft_consensus.cc:740] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.700875 9714 consensus_queue.cc:260] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:55.700984 9714 ts_tablet_manager.cc:1434] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:55.701033 9201 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36406
I20251026 09:11:55.701263 9788 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:55.701450 3128 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:55.701503 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "226d829f3750486bbe2e8ca2667b36df"
format_stamp: "Formatted at 2025-10-26 09:11:47 on dist-test-slave-w8k0"
I20251026 09:11:55.701527 9775 heartbeater.cc:499] Master 127.3.14.62:45323 was elected leader, sending a full tablet report...
I20251026 09:11:55.701556 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:55.707875 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:55.708204 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:55.708396 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:55.710057 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20251026 09:11:55.710131 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20251026 09:11:55.710629 9795 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap starting.
I20251026 09:11:55.711194 3128 ts_tablet_manager.cc:616] Registered 4 tablets
I20251026 09:11:55.711237 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251026 09:11:55.711911 9795 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:55.712097 9795 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap complete.
I20251026 09:11:55.712219 9795 ts_tablet_manager.cc:1403] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:55.712438 9795 raft_consensus.cc:359] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.712541 9795 raft_consensus.cc:740] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.712653 9795 consensus_queue.cc:260] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.712838 9795 ts_tablet_manager.cc:1434] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:55.712922 9795 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap starting.
I20251026 09:11:55.714185 9795 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:55.714380 9795 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap complete.
I20251026 09:11:55.714497 9795 ts_tablet_manager.cc:1403] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:55.714679 9795 raft_consensus.cc:359] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.714740 9795 raft_consensus.cc:740] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.714784 9795 consensus_queue.cc:260] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.714921 9795 ts_tablet_manager.cc:1434] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:55.714982 9795 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap starting.
I20251026 09:11:55.715245 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:34513
I20251026 09:11:55.715456 9855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:34513 every 8 connection(s)
I20251026 09:11:55.716252 9795 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:55.716450 9795 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: Bootstrap complete.
I20251026 09:11:55.716578 9795 ts_tablet_manager.cc:1403] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:55.716764 9795 raft_consensus.cc:359] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.716853 9795 raft_consensus.cc:740] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.716925 9795 consensus_queue.cc:260] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:55.717031 9795 ts_tablet_manager.cc:1434] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:55.717108 9795 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: Bootstrap starting.
I20251026 09:11:55.718333 9795 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:55.718546 9795 tablet_bootstrap.cc:492] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: Bootstrap complete.
I20251026 09:11:55.718665 9795 ts_tablet_manager.cc:1403] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:55.718885 9795 raft_consensus.cc:359] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:55.718962 9795 raft_consensus.cc:740] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Initialized, Role: FOLLOWER
I20251026 09:11:55.719029 9795 consensus_queue.cc:260] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:55.719146 9795 ts_tablet_manager.cc:1434] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:11:55.719539 9856 heartbeater.cc:344] Connected to a master server at 127.3.14.62:45323
I20251026 09:11:55.719604 9856 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:55.719703 9856 heartbeater.cc:507] Master 127.3.14.62:45323 requested a full tablet report, sending...
I20251026 09:11:55.719993 9201 ts_manager.cc:194] Re-registered known tserver with Master: 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:55.720345 9201 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36414
I20251026 09:11:55.720791 9856 heartbeater.cc:499] Master 127.3.14.62:45323 was elected leader, sending a full tablet report...
W20251026 09:11:56.091435 9249 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Replica d49cd53f5dd64a25a511de611a4906ff is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } }
W20251026 09:11:56.092154 9249 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e97341eb7bf94ce6901ad16486ba4138, TS 244142e66bf8451f9c8d3c584c6c8c8d -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:56.092223 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet e97341eb7bf94ce6901ad16486ba4138, TS 244142e66bf8451f9c8d3c584c6c8c8d -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:56.095769 9249 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3b8f80e14b9a4606b0ee24503fb879c9, TS 226d829f3750486bbe2e8ca2667b36df -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:56.095863 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 3b8f80e14b9a4606b0ee24503fb879c9, TS 226d829f3750486bbe2e8ca2667b36df -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:56.096308 9249 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1a0d9035730140a4b624f225743a2ae5, TS d49cd53f5dd64a25a511de611a4906ff -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:56.096361 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 1a0d9035730140a4b624f225743a2ae5, TS d49cd53f5dd64a25a511de611a4906ff -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:56.832259 9864 raft_consensus.cc:493] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:56.832376 9864 raft_consensus.cc:515] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:56.832612 9864 leader_election.cc:290] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:56.835740 9864 raft_consensus.cc:493] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:56.835873 9864 raft_consensus.cc:515] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:56.836038 9864 leader_election.cc:290] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:56.836510 9748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e97341eb7bf94ce6901ad16486ba4138" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff" is_pre_election: true
I20251026 09:11:56.836546 9747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbef2b4cd5ab404e89c48a8452f2fe1c" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff" is_pre_election: true
I20251026 09:11:56.836642 9748 raft_consensus.cc:2410] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 244142e66bf8451f9c8d3c584c6c8c8d for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251026 09:11:56.836668 9747 raft_consensus.cc:2468] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 1.
I20251026 09:11:56.836571 9828 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbef2b4cd5ab404e89c48a8452f2fe1c" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df" is_pre_election: true
I20251026 09:11:56.836562 9829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e97341eb7bf94ce6901ad16486ba4138" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df" is_pre_election: true
I20251026 09:11:56.836767 9828 raft_consensus.cc:2468] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 1.
I20251026 09:11:56.836798 9829 raft_consensus.cc:2468] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 1.
I20251026 09:11:56.836900 9627 leader_election.cc:304] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244142e66bf8451f9c8d3c584c6c8c8d, d49cd53f5dd64a25a511de611a4906ff; no voters:
I20251026 09:11:56.836995 9628 leader_election.cc:304] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, 244142e66bf8451f9c8d3c584c6c8c8d; no voters: d49cd53f5dd64a25a511de611a4906ff
I20251026 09:11:56.837018 9864 raft_consensus.cc:2804] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 09:11:56.837069 9864 raft_consensus.cc:493] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:56.837091 9864 raft_consensus.cc:3060] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:56.837244 9869 raft_consensus.cc:2804] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 09:11:56.837285 9869 raft_consensus.cc:493] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:56.837311 9869 raft_consensus.cc:3060] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:56.837934 9864 raft_consensus.cc:515] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:56.838076 9864 leader_election.cc:290] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 2 election: Requested vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:56.838040 9869 raft_consensus.cc:515] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:56.838224 9869 leader_election.cc:290] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 2 election: Requested vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:56.838270 9829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbef2b4cd5ab404e89c48a8452f2fe1c" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df"
I20251026 09:11:56.838294 9747 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e97341eb7bf94ce6901ad16486ba4138" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff"
I20251026 09:11:56.838351 9747 raft_consensus.cc:3060] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:56.838338 9829 raft_consensus.cc:3060] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:56.838375 9748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbef2b4cd5ab404e89c48a8452f2fe1c" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff"
I20251026 09:11:56.838361 9828 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e97341eb7bf94ce6901ad16486ba4138" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df"
I20251026 09:11:56.838474 9828 raft_consensus.cc:3060] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:56.838435 9748 raft_consensus.cc:3060] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:56.839354 9829 raft_consensus.cc:2468] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 2.
I20251026 09:11:56.839362 9747 raft_consensus.cc:2410] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 244142e66bf8451f9c8d3c584c6c8c8d for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251026 09:11:56.839443 9828 raft_consensus.cc:2468] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 2.
I20251026 09:11:56.839511 9748 raft_consensus.cc:2468] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 2.
I20251026 09:11:56.839576 9628 leader_election.cc:304] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, 244142e66bf8451f9c8d3c584c6c8c8d; no voters:
I20251026 09:11:56.839643 9628 leader_election.cc:304] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, 244142e66bf8451f9c8d3c584c6c8c8d; no voters: d49cd53f5dd64a25a511de611a4906ff
I20251026 09:11:56.839664 9869 raft_consensus.cc:2804] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:11:56.839717 9864 raft_consensus.cc:2804] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:11:56.839876 9864 raft_consensus.cc:697] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 LEADER]: Becoming Leader. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Running, Role: LEADER
I20251026 09:11:56.839876 9869 raft_consensus.cc:697] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 LEADER]: Becoming Leader. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Running, Role: LEADER
I20251026 09:11:56.839960 9869 consensus_queue.cc:237] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } }
I20251026 09:11:56.839960 9864 consensus_queue.cc:237] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:56.840674 9201 catalog_manager.cc:5649] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d reported cstate change: term changed from 1 to 2, leader changed from d49cd53f5dd64a25a511de611a4906ff (127.3.14.2) to 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1). New cstate: current_term: 2 leader_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:56.840780 9201 catalog_manager.cc:5649] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } health_report { overall_health: HEALTHY } } }
I20251026 09:11:56.924081 9878 raft_consensus.cc:493] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:56.924194 9878 raft_consensus.cc:515] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:56.924382 9878 leader_election.cc:290] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:56.927249 9828 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8f80e14b9a4606b0ee24503fb879c9" candidate_uuid: "d49cd53f5dd64a25a511de611a4906ff" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df" is_pre_election: true
I20251026 09:11:56.927356 9666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8f80e14b9a4606b0ee24503fb879c9" candidate_uuid: "d49cd53f5dd64a25a511de611a4906ff" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" is_pre_election: true
I20251026 09:11:56.927414 9828 raft_consensus.cc:2468] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49cd53f5dd64a25a511de611a4906ff in term 1.
I20251026 09:11:56.927471 9666 raft_consensus.cc:2410] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d49cd53f5dd64a25a511de611a4906ff for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251026 09:11:56.927595 9710 leader_election.cc:304] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, d49cd53f5dd64a25a511de611a4906ff; no voters:
I20251026 09:11:56.927690 9878 raft_consensus.cc:2804] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 09:11:56.927771 9878 raft_consensus.cc:493] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:56.927793 9878 raft_consensus.cc:3060] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:56.928484 9878 raft_consensus.cc:515] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:56.928608 9878 leader_election.cc:290] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [CANDIDATE]: Term 2 election: Requested vote from peers 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:56.928787 9666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8f80e14b9a4606b0ee24503fb879c9" candidate_uuid: "d49cd53f5dd64a25a511de611a4906ff" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244142e66bf8451f9c8d3c584c6c8c8d"
I20251026 09:11:56.928843 9666 raft_consensus.cc:3060] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:56.928824 9828 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8f80e14b9a4606b0ee24503fb879c9" candidate_uuid: "d49cd53f5dd64a25a511de611a4906ff" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df"
I20251026 09:11:56.928939 9828 raft_consensus.cc:3060] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:56.929526 9666 raft_consensus.cc:2410] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d49cd53f5dd64a25a511de611a4906ff for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251026 09:11:56.929587 9828 raft_consensus.cc:2468] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d49cd53f5dd64a25a511de611a4906ff in term 2.
I20251026 09:11:56.929733 9710 leader_election.cc:304] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, d49cd53f5dd64a25a511de611a4906ff; no voters: 244142e66bf8451f9c8d3c584c6c8c8d
I20251026 09:11:56.929831 9878 raft_consensus.cc:2804] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:11:56.929998 9878 raft_consensus.cc:697] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 2 LEADER]: Becoming Leader. State: Replica: d49cd53f5dd64a25a511de611a4906ff, State: Running, Role: LEADER
I20251026 09:11:56.930063 9878 consensus_queue.cc:237] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:56.930626 9201 catalog_manager.cc:5649] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff reported cstate change: term changed from 1 to 2, leader changed from 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1) to d49cd53f5dd64a25a511de611a4906ff (127.3.14.2). New cstate: current_term: 2 leader_uuid: "d49cd53f5dd64a25a511de611a4906ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: UNKNOWN } } }
W20251026 09:11:57.097577 9249 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20251026 09:11:57.098232 9249 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3b8f80e14b9a4606b0ee24503fb879c9, TS 244142e66bf8451f9c8d3c584c6c8c8d -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:57.098302 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 3b8f80e14b9a4606b0ee24503fb879c9, TS 244142e66bf8451f9c8d3c584c6c8c8d -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:57.098901 9249 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e97341eb7bf94ce6901ad16486ba4138, TS 226d829f3750486bbe2e8ca2667b36df -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:57.098966 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet e97341eb7bf94ce6901ad16486ba4138, TS 226d829f3750486bbe2e8ca2667b36df -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:57.099462 9249 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1a0d9035730140a4b624f225743a2ae5, TS d49cd53f5dd64a25a511de611a4906ff -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
W20251026 09:11:57.099511 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 1a0d9035730140a4b624f225743a2ae5, TS d49cd53f5dd64a25a511de611a4906ff -> TS 867eac79b8c74013b9e5cad031815cca move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:57.240245 9748 raft_consensus.cc:1275] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20251026 09:11:57.240334 9748 pending_rounds.cc:85] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Aborting all ops after (but not including) 0
I20251026 09:11:57.240351 9748 pending_rounds.cc:107] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20251026 09:11:57.240365 9748 raft_consensus.cc:2889] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 09:11:57.240608 9869 consensus_queue.cc:1048] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:57.241869 9828 raft_consensus.cc:1275] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 09:11:57.242172 9869 consensus_queue.cc:1048] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:57.260390 9887 raft_consensus.cc:493] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:57.260578 9887 raft_consensus.cc:515] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:57.260901 9887 leader_election.cc:290] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213), d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035)
I20251026 09:11:57.264264 9748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "226d829f3750486bbe2e8ca2667b36df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff" is_pre_election: true
I20251026 09:11:57.264343 9666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "226d829f3750486bbe2e8ca2667b36df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" is_pre_election: true
I20251026 09:11:57.264415 9748 raft_consensus.cc:2468] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226d829f3750486bbe2e8ca2667b36df in term 1.
I20251026 09:11:57.264428 9666 raft_consensus.cc:2410] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 226d829f3750486bbe2e8ca2667b36df for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251026 09:11:57.264596 9790 leader_election.cc:304] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, d49cd53f5dd64a25a511de611a4906ff; no voters:
I20251026 09:11:57.264709 9887 raft_consensus.cc:2804] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 09:11:57.264756 9887 raft_consensus.cc:493] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:57.264771 9887 raft_consensus.cc:3060] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:57.265432 9887 raft_consensus.cc:515] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:57.265587 9887 leader_election.cc:290] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [CANDIDATE]: Term 2 election: Requested vote from peers 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213), d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035)
I20251026 09:11:57.265772 9666 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "226d829f3750486bbe2e8ca2667b36df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244142e66bf8451f9c8d3c584c6c8c8d"
I20251026 09:11:57.265811 9748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "226d829f3750486bbe2e8ca2667b36df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff"
I20251026 09:11:57.265893 9666 raft_consensus.cc:3060] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:57.265916 9748 raft_consensus.cc:3060] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 1 FOLLOWER]: Advancing to term 2
I20251026 09:11:57.266654 9748 raft_consensus.cc:2468] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226d829f3750486bbe2e8ca2667b36df in term 2.
I20251026 09:11:57.266680 9666 raft_consensus.cc:2410] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 226d829f3750486bbe2e8ca2667b36df for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251026 09:11:57.266835 9790 leader_election.cc:304] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226d829f3750486bbe2e8ca2667b36df, d49cd53f5dd64a25a511de611a4906ff; no voters:
I20251026 09:11:57.266953 9887 raft_consensus.cc:2804] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Leader election won for term 2
I20251026 09:11:57.267102 9887 raft_consensus.cc:697] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 2 LEADER]: Becoming Leader. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Running, Role: LEADER
I20251026 09:11:57.267163 9887 consensus_queue.cc:237] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:57.267797 9201 catalog_manager.cc:5649] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df reported cstate change: term changed from 1 to 2, leader changed from 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1) to 226d829f3750486bbe2e8ca2667b36df (127.3.14.3). New cstate: current_term: 2 leader_uuid: "226d829f3750486bbe2e8ca2667b36df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: HEALTHY } } }
I20251026 09:11:57.301146 9828 raft_consensus.cc:1275] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:11:57.301440 9869 consensus_queue.cc:1048] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:57.303267 9748 raft_consensus.cc:1275] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:11:57.303536 9869 consensus_queue.cc:1048] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:57.370553 9828 raft_consensus.cc:1275] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Refusing update from remote peer d49cd53f5dd64a25a511de611a4906ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 09:11:57.370940 9882 consensus_queue.cc:1048] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:57.372372 9666 raft_consensus.cc:1275] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Refusing update from remote peer d49cd53f5dd64a25a511de611a4906ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20251026 09:11:57.372450 9666 pending_rounds.cc:85] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Aborting all ops after (but not including) 0
I20251026 09:11:57.372475 9666 pending_rounds.cc:107] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20251026 09:11:57.372500 9666 raft_consensus.cc:2889] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 09:11:57.372779 9897 consensus_queue.cc:1048] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:57.433914 9869 raft_consensus.cc:493] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:11:57.434044 9869 raft_consensus.cc:515] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:57.434340 9869 leader_election.cc:290] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:57.434558 9828 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df" is_pre_election: true
I20251026 09:11:57.434558 9748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff" is_pre_election: true
I20251026 09:11:57.434708 9748 raft_consensus.cc:2468] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 2.
I20251026 09:11:57.434873 9627 leader_election.cc:304] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244142e66bf8451f9c8d3c584c6c8c8d, d49cd53f5dd64a25a511de611a4906ff; no voters: 226d829f3750486bbe2e8ca2667b36df
I20251026 09:11:57.434984 9869 raft_consensus.cc:2804] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 09:11:57.435029 9869 raft_consensus.cc:493] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:11:57.435048 9869 raft_consensus.cc:3060] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:11:57.435945 9869 raft_consensus.cc:515] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:57.436087 9869 leader_election.cc:290] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 3 election: Requested vote from peers d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035), 226d829f3750486bbe2e8ca2667b36df (127.3.14.3:34513)
I20251026 09:11:57.436270 9748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d49cd53f5dd64a25a511de611a4906ff"
I20251026 09:11:57.436270 9828 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0d9035730140a4b624f225743a2ae5" candidate_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "226d829f3750486bbe2e8ca2667b36df"
I20251026 09:11:57.436386 9748 raft_consensus.cc:3060] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:11:57.437158 9748 raft_consensus.cc:2468] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244142e66bf8451f9c8d3c584c6c8c8d in term 3.
I20251026 09:11:57.437386 9627 leader_election.cc:304] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244142e66bf8451f9c8d3c584c6c8c8d, d49cd53f5dd64a25a511de611a4906ff; no voters: 226d829f3750486bbe2e8ca2667b36df
I20251026 09:11:57.437490 9869 raft_consensus.cc:2804] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 3 FOLLOWER]: Leader election won for term 3
I20251026 09:11:57.437551 9869 raft_consensus.cc:697] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 3 LEADER]: Becoming Leader. State: Replica: 244142e66bf8451f9c8d3c584c6c8c8d, State: Running, Role: LEADER
I20251026 09:11:57.437614 9869 consensus_queue.cc:237] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:57.438138 9202 catalog_manager.cc:5649] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d reported cstate change: term changed from 2 to 3, leader changed from 226d829f3750486bbe2e8ca2667b36df (127.3.14.3) to 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1). New cstate: current_term: 3 leader_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:57.698827 9666 raft_consensus.cc:1240] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 3 LEADER]: Rejecting Update request from peer 226d829f3750486bbe2e8ca2667b36df for earlier term 2. Current term is 3. Ops: []
I20251026 09:11:57.699188 9892 consensus_queue.cc:1059] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:57.699316 9892 consensus_queue.cc:1243] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [LEADER]: Peer 244142e66bf8451f9c8d3c584c6c8c8d log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251026 09:11:57.699399 9887 raft_consensus.cc:3055] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 2 LEADER]: Stepping down as leader of term 2
I20251026 09:11:57.699471 9887 raft_consensus.cc:740] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 226d829f3750486bbe2e8ca2667b36df, State: Running, Role: LEADER
I20251026 09:11:57.699560 9887 consensus_queue.cc:260] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } }
I20251026 09:11:57.699694 9887 raft_consensus.cc:3060] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Advancing to term 3
I20251026 09:11:57.882292 9828 raft_consensus.cc:1275] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 3 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 09:11:57.882650 9864 consensus_queue.cc:1048] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:57.882797 9864 consensus_queue.cc:1243] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Peer 226d829f3750486bbe2e8ca2667b36df log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251026 09:11:57.883139 9828 pending_rounds.cc:85] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Aborting all ops after (but not including) 0
I20251026 09:11:57.883196 9828 pending_rounds.cc:107] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20251026 09:11:57.883212 9828 raft_consensus.cc:2889] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 09:11:57.884421 9748 raft_consensus.cc:1275] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 3 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 09:11:57.884667 9870 consensus_queue.cc:1048] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:11:58.100499 9748 consensus_queue.cc:237] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } attrs { replace: true } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } }
I20251026 09:11:58.101168 9666 raft_consensus.cc:1275] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Refusing update from remote peer d49cd53f5dd64a25a511de611a4906ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:11:58.101226 9828 raft_consensus.cc:1275] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Refusing update from remote peer d49cd53f5dd64a25a511de611a4906ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:11:58.101428 9882 consensus_queue.cc:1048] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:58.101589 9904 consensus_queue.cc:1048] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:58.102178 9882 raft_consensus.cc:2955] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } attrs { replace: true } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.102293 9666 raft_consensus.cc:2955] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } attrs { replace: true } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.103602 9828 raft_consensus.cc:2955] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } attrs { replace: true } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.103623 9201 catalog_manager.cc:5649] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff reported cstate change: config changed from index -1 to 2, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New cstate: current_term: 2 leader_uuid: "d49cd53f5dd64a25a511de611a4906ff" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:11:58.104091 9666 consensus_queue.cc:237] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } attrs { replace: true } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } }
I20251026 09:11:58.104593 9829 raft_consensus.cc:1275] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 3 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 09:11:58.104748 9748 raft_consensus.cc:1275] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 3 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 09:11:58.105007 9864 consensus_queue.cc:1048] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251026 09:11:58.105069 9711 consensus_peers.cc:597] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff -> Peer 867eac79b8c74013b9e5cad031815cca (127.3.14.4:34947): Couldn't send request to peer 867eac79b8c74013b9e5cad031815cca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b8f80e14b9a4606b0ee24503fb879c9. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:58.105497 9864 consensus_queue.cc:1048] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:11:58.105612 9870 raft_consensus.cc:2955] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } attrs { replace: true } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.106037 9829 raft_consensus.cc:2955] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } attrs { replace: true } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.106901 9748 raft_consensus.cc:2955] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } attrs { replace: true } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.106933 9666 consensus_queue.cc:237] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } attrs { replace: true } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } }
I20251026 09:11:58.107777 9200 catalog_manager.cc:5649] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df reported cstate change: config changed from index -1 to 3, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New cstate: current_term: 3 leader_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } attrs { replace: true } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
W20251026 09:11:58.107993 9629 consensus_peers.cc:597] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d -> Peer 867eac79b8c74013b9e5cad031815cca (127.3.14.4:34947): Couldn't send request to peer 867eac79b8c74013b9e5cad031815cca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a0d9035730140a4b624f225743a2ae5. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:58.108080 9747 raft_consensus.cc:1275] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 09:11:58.108188 9629 consensus_peers.cc:597] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d -> Peer 867eac79b8c74013b9e5cad031815cca (127.3.14.4:34947): Couldn't send request to peer 867eac79b8c74013b9e5cad031815cca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e97341eb7bf94ce6901ad16486ba4138. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:11:58.108232 9864 consensus_queue.cc:1048] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:58.108822 9870 raft_consensus.cc:2955] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } attrs { replace: true } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.109118 9829 raft_consensus.cc:1275] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Refusing update from remote peer 244142e66bf8451f9c8d3c584c6c8c8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 09:11:58.109215 9748 raft_consensus.cc:2955] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } attrs { replace: true } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.109288 9869 consensus_queue.cc:1048] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:11:58.109602 9829 raft_consensus.cc:2955] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } attrs { replace: true } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.110332 9202 catalog_manager.cc:5649] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff reported cstate change: config changed from index -1 to 2, NON_VOTER 867eac79b8c74013b9e5cad031815cca (127.3.14.4) added. New cstate: current_term: 2 leader_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } attrs { replace: true } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } } }
I20251026 09:11:58.551151 9912 ts_tablet_manager.cc:933] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: Initiating tablet copy from peer 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213)
I20251026 09:11:58.551388 9912 tablet_copy_client.cc:323] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.1:37213
I20251026 09:11:58.554774 9677 tablet_copy_service.cc:140] P 244142e66bf8451f9c8d3c584c6c8c8d: Received BeginTabletCopySession request for tablet 1a0d9035730140a4b624f225743a2ae5 from peer 867eac79b8c74013b9e5cad031815cca ({username='slave'} at 127.0.0.1:54530)
I20251026 09:11:58.554859 9677 tablet_copy_service.cc:161] P 244142e66bf8451f9c8d3c584c6c8c8d: Beginning new tablet copy session on tablet 1a0d9035730140a4b624f225743a2ae5 from peer 867eac79b8c74013b9e5cad031815cca at {username='slave'} at 127.0.0.1:54530: session id = 867eac79b8c74013b9e5cad031815cca-1a0d9035730140a4b624f225743a2ae5
I20251026 09:11:58.555526 9677 tablet_copy_source_session.cc:215] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:11:58.555894 9912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0d9035730140a4b624f225743a2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:58.557397 9912 tablet_copy_client.cc:806] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Starting download of 0 data blocks...
I20251026 09:11:58.557531 9912 tablet_copy_client.cc:670] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Starting download of 1 WAL segments...
I20251026 09:11:58.558058 9912 tablet_copy_client.cc:538] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:58.558868 9912 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: Bootstrap starting.
I20251026 09:11:58.560565 9912 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:58.560782 9912 tablet_bootstrap.cc:492] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: Bootstrap complete.
I20251026 09:11:58.560937 9912 ts_tablet_manager.cc:1403] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.003s
I20251026 09:11:58.561148 9912 raft_consensus.cc:359] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } attrs { replace: true } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } }
I20251026 09:11:58.561249 9912 raft_consensus.cc:740] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 867eac79b8c74013b9e5cad031815cca, State: Initialized, Role: LEARNER
I20251026 09:11:58.561316 9912 consensus_queue.cc:260] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } attrs { replace: true } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } }
I20251026 09:11:58.561547 9912 ts_tablet_manager.cc:1434] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:58.561789 9677 tablet_copy_service.cc:342] P 244142e66bf8451f9c8d3c584c6c8c8d: Request end of tablet copy session 867eac79b8c74013b9e5cad031815cca-1a0d9035730140a4b624f225743a2ae5 received from {username='slave'} at 127.0.0.1:54530
I20251026 09:11:58.561841 9677 tablet_copy_service.cc:434] P 244142e66bf8451f9c8d3c584c6c8c8d: ending tablet copy session 867eac79b8c74013b9e5cad031815cca-1a0d9035730140a4b624f225743a2ae5 on tablet 1a0d9035730140a4b624f225743a2ae5 with peer 867eac79b8c74013b9e5cad031815cca
I20251026 09:11:58.594339 9912 ts_tablet_manager.cc:933] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: Initiating tablet copy from peer d49cd53f5dd64a25a511de611a4906ff (127.3.14.2:39035)
I20251026 09:11:58.594558 9912 tablet_copy_client.cc:323] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:39035
I20251026 09:11:58.598223 9759 tablet_copy_service.cc:140] P d49cd53f5dd64a25a511de611a4906ff: Received BeginTabletCopySession request for tablet 3b8f80e14b9a4606b0ee24503fb879c9 from peer 867eac79b8c74013b9e5cad031815cca ({username='slave'} at 127.0.0.1:37574)
I20251026 09:11:58.598325 9759 tablet_copy_service.cc:161] P d49cd53f5dd64a25a511de611a4906ff: Beginning new tablet copy session on tablet 3b8f80e14b9a4606b0ee24503fb879c9 from peer 867eac79b8c74013b9e5cad031815cca at {username='slave'} at 127.0.0.1:37574: session id = 867eac79b8c74013b9e5cad031815cca-3b8f80e14b9a4606b0ee24503fb879c9
I20251026 09:11:58.598821 9759 tablet_copy_source_session.cc:215] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:11:58.599261 9912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8f80e14b9a4606b0ee24503fb879c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:58.600785 9912 tablet_copy_client.cc:806] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Starting download of 0 data blocks...
I20251026 09:11:58.600924 9912 tablet_copy_client.cc:670] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Starting download of 1 WAL segments...
I20251026 09:11:58.601572 9912 tablet_copy_client.cc:538] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:58.602463 9912 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: Bootstrap starting.
I20251026 09:11:58.604071 9912 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:58.604245 9912 tablet_bootstrap.cc:492] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: Bootstrap complete.
I20251026 09:11:58.604357 9912 ts_tablet_manager.cc:1403] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251026 09:11:58.604523 9912 raft_consensus.cc:359] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } attrs { replace: true } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } }
I20251026 09:11:58.604596 9912 raft_consensus.cc:740] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 867eac79b8c74013b9e5cad031815cca, State: Initialized, Role: LEARNER
I20251026 09:11:58.604641 9912 consensus_queue.cc:260] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } attrs { replace: true } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } }
I20251026 09:11:58.604831 9912 ts_tablet_manager.cc:1434] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:58.605037 9759 tablet_copy_service.cc:342] P d49cd53f5dd64a25a511de611a4906ff: Request end of tablet copy session 867eac79b8c74013b9e5cad031815cca-3b8f80e14b9a4606b0ee24503fb879c9 received from {username='slave'} at 127.0.0.1:37574
I20251026 09:11:58.605090 9759 tablet_copy_service.cc:434] P d49cd53f5dd64a25a511de611a4906ff: ending tablet copy session 867eac79b8c74013b9e5cad031815cca-3b8f80e14b9a4606b0ee24503fb879c9 on tablet 3b8f80e14b9a4606b0ee24503fb879c9 with peer 867eac79b8c74013b9e5cad031815cca
I20251026 09:11:58.693547 9912 ts_tablet_manager.cc:933] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: Initiating tablet copy from peer 244142e66bf8451f9c8d3c584c6c8c8d (127.3.14.1:37213)
I20251026 09:11:58.693773 9912 tablet_copy_client.cc:323] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.1:37213
I20251026 09:11:58.694028 9677 tablet_copy_service.cc:140] P 244142e66bf8451f9c8d3c584c6c8c8d: Received BeginTabletCopySession request for tablet e97341eb7bf94ce6901ad16486ba4138 from peer 867eac79b8c74013b9e5cad031815cca ({username='slave'} at 127.0.0.1:54530)
I20251026 09:11:58.694092 9677 tablet_copy_service.cc:161] P 244142e66bf8451f9c8d3c584c6c8c8d: Beginning new tablet copy session on tablet e97341eb7bf94ce6901ad16486ba4138 from peer 867eac79b8c74013b9e5cad031815cca at {username='slave'} at 127.0.0.1:54530: session id = 867eac79b8c74013b9e5cad031815cca-e97341eb7bf94ce6901ad16486ba4138
I20251026 09:11:58.694578 9677 tablet_copy_source_session.cc:215] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:11:58.694955 9912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e97341eb7bf94ce6901ad16486ba4138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:58.696280 9912 tablet_copy_client.cc:806] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Starting download of 0 data blocks...
I20251026 09:11:58.696395 9912 tablet_copy_client.cc:670] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Starting download of 1 WAL segments...
I20251026 09:11:58.696972 9912 tablet_copy_client.cc:538] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:11:58.697571 9912 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: Bootstrap starting.
I20251026 09:11:58.698786 9912 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:11:58.698975 9912 tablet_bootstrap.cc:492] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: Bootstrap complete.
I20251026 09:11:58.699085 9912 ts_tablet_manager.cc:1403] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:11:58.699257 9912 raft_consensus.cc:359] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } attrs { replace: true } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } }
I20251026 09:11:58.699358 9912 raft_consensus.cc:740] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 867eac79b8c74013b9e5cad031815cca, State: Initialized, Role: LEARNER
I20251026 09:11:58.699414 9912 consensus_queue.cc:260] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "244142e66bf8451f9c8d3c584c6c8c8d" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 37213 } } peers { permanent_uuid: "d49cd53f5dd64a25a511de611a4906ff" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 39035 } attrs { replace: true } } peers { permanent_uuid: "226d829f3750486bbe2e8ca2667b36df" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34513 } } peers { permanent_uuid: "867eac79b8c74013b9e5cad031815cca" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 34947 } attrs { promote: true } }
I20251026 09:11:58.699569 9912 ts_tablet_manager.cc:1434] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:11:58.699769 9677 tablet_copy_service.cc:342] P 244142e66bf8451f9c8d3c584c6c8c8d: Request end of tablet copy session 867eac79b8c74013b9e5cad031815cca-e97341eb7bf94ce6901ad16486ba4138 received from {username='slave'} at 127.0.0.1:54530
I20251026 09:11:58.699829 9677 tablet_copy_service.cc:434] P 244142e66bf8451f9c8d3c584c6c8c8d: ending tablet copy session 867eac79b8c74013b9e5cad031815cca-e97341eb7bf94ce6901ad16486ba4138 on tablet e97341eb7bf94ce6901ad16486ba4138 with peer 867eac79b8c74013b9e5cad031815cca
I20251026 09:11:59.009063 9576 raft_consensus.cc:1217] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20251026 09:11:59.034230 9576 raft_consensus.cc:1217] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20251026 09:11:59.228951 3128 tablet_server.cc:178] TabletServer@127.3.14.1:37213 shutting down...
W20251026 09:11:59.229501 9247 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 11 similar messages]
W20251026 09:11:59.230757 9249 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 09:11:59.230841 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 09:11:59.231770 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:59.231918 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.231992 3128 raft_consensus.cc:2243] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 LEADER]: Raft consensus shutting down.
I20251026 09:11:59.232086 3128 raft_consensus.cc:2272] T e97341eb7bf94ce6901ad16486ba4138 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.232327 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.232370 3128 raft_consensus.cc:2243] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 3 LEADER]: Raft consensus shutting down.
I20251026 09:11:59.232417 3128 raft_consensus.cc:2272] T 1a0d9035730140a4b624f225743a2ae5 P 244142e66bf8451f9c8d3c584c6c8c8d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.232538 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.232571 3128 raft_consensus.cc:2243] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 LEADER]: Raft consensus shutting down.
I20251026 09:11:59.232620 3128 raft_consensus.cc:2272] T cbef2b4cd5ab404e89c48a8452f2fe1c P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.232741 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.232774 3128 raft_consensus.cc:2243] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:59.232797 3128 raft_consensus.cc:2272] T 3b8f80e14b9a4606b0ee24503fb879c9 P 244142e66bf8451f9c8d3c584c6c8c8d [term 2 FOLLOWER]: Raft consensus is shut down!
W20251026 09:11:59.233031 9249 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 09:11:59.233065 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 09:11:59.235008 3128 tablet_server.cc:195] TabletServer@127.3.14.1:37213 shutdown complete.
I20251026 09:11:59.236424 3128 tablet_server.cc:178] TabletServer@127.3.14.2:39035 shutting down...
W20251026 09:11:59.237286 9249 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 09:11:59.237337 9249 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 09:11:59.239143 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:59.239262 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.239321 3128 raft_consensus.cc:2243] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:59.239414 3128 raft_consensus.cc:2272] T e97341eb7bf94ce6901ad16486ba4138 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.239594 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.239631 3128 raft_consensus.cc:2243] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:59.239674 3128 raft_consensus.cc:2272] T 1a0d9035730140a4b624f225743a2ae5 P d49cd53f5dd64a25a511de611a4906ff [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.239782 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.239821 3128 raft_consensus.cc:2243] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:59.239856 3128 raft_consensus.cc:2272] T cbef2b4cd5ab404e89c48a8452f2fe1c P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.239979 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.240010 3128 raft_consensus.cc:2243] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 2 LEADER]: Raft consensus shutting down.
I20251026 09:11:59.240082 3128 raft_consensus.cc:2272] T 3b8f80e14b9a4606b0ee24503fb879c9 P d49cd53f5dd64a25a511de611a4906ff [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.252199 3128 tablet_server.cc:195] TabletServer@127.3.14.2:39035 shutdown complete.
I20251026 09:11:59.253643 3128 tablet_server.cc:178] TabletServer@127.3.14.3:34513 shutting down...
I20251026 09:11:59.257603 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:59.257758 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.257828 3128 raft_consensus.cc:2243] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:59.257895 3128 raft_consensus.cc:2272] T e97341eb7bf94ce6901ad16486ba4138 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.258090 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.258131 3128 raft_consensus.cc:2243] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:59.258162 3128 raft_consensus.cc:2272] T 1a0d9035730140a4b624f225743a2ae5 P 226d829f3750486bbe2e8ca2667b36df [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.258287 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.258325 3128 raft_consensus.cc:2243] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:59.258363 3128 raft_consensus.cc:2272] T cbef2b4cd5ab404e89c48a8452f2fe1c P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.258479 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:11:59.258512 3128 raft_consensus.cc:2243] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 09:11:59.258553 3128 raft_consensus.cc:2272] T 3b8f80e14b9a4606b0ee24503fb879c9 P 226d829f3750486bbe2e8ca2667b36df [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.270339 3128 tablet_server.cc:195] TabletServer@127.3.14.3:34513 shutdown complete.
I20251026 09:11:59.271566 3128 tablet_server.cc:178] TabletServer@127.3.14.4:0 shutting down...
I20251026 09:11:59.275067 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:11:59.275271 3128 tablet_replica.cc:333] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca: stopping tablet replica
I20251026 09:11:59.275409 3128 raft_consensus.cc:2243] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca [term 2 LEARNER]: Raft consensus shutting down.
I20251026 09:11:59.275467 3128 raft_consensus.cc:2272] T e97341eb7bf94ce6901ad16486ba4138 P 867eac79b8c74013b9e5cad031815cca [term 2 LEARNER]: Raft consensus is shut down!
I20251026 09:11:59.275667 3128 tablet_replica.cc:333] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca: stopping tablet replica
I20251026 09:11:59.275704 3128 raft_consensus.cc:2243] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca [term 3 LEARNER]: Raft consensus shutting down.
I20251026 09:11:59.275732 3128 raft_consensus.cc:2272] T 1a0d9035730140a4b624f225743a2ae5 P 867eac79b8c74013b9e5cad031815cca [term 3 LEARNER]: Raft consensus is shut down!
I20251026 09:11:59.275846 3128 tablet_replica.cc:333] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca: stopping tablet replica
I20251026 09:11:59.275880 3128 raft_consensus.cc:2243] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca [term 2 LEARNER]: Raft consensus shutting down.
I20251026 09:11:59.275920 3128 raft_consensus.cc:2272] T 3b8f80e14b9a4606b0ee24503fb879c9 P 867eac79b8c74013b9e5cad031815cca [term 2 LEARNER]: Raft consensus is shut down!
I20251026 09:11:59.287945 3128 tablet_server.cc:195] TabletServer@127.3.14.4:0 shutdown complete.
I20251026 09:11:59.289110 3128 master.cc:561] Master@127.3.14.62:45323 shutting down...
I20251026 09:11:59.291731 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:11:59.291831 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:11:59.291852 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 09c6d9e2dde9462e9b86200c3ffc1a48: stopping tablet replica
I20251026 09:11:59.303907 3128 master.cc:583] Master@127.3.14.62:45323 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (12261 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20251026 09:11:59.311167 3128 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.14.62:35803
I20251026 09:11:59.311368 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:59.312741 9928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:59.312824 9925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:59.312837 9926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:59.312955 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:59.313056 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:59.313081 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:59.313094 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469919313094 us; error 0 us; skew 500 ppm
I20251026 09:11:59.313666 3128 webserver.cc:492] Webserver started at http://127.3.14.62:34389/ using document root <none> and password file <none>
I20251026 09:11:59.313769 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:59.313820 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:59.313871 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:59.314162 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/master-0-root/instance:
uuid: "9ba52672e5fe4bbca8dd8cc52e42149f"
format_stamp: "Formatted at 2025-10-26 09:11:59 on dist-test-slave-w8k0"
I20251026 09:11:59.315070 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:59.315614 9933 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.315739 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.315796 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/master-0-root
uuid: "9ba52672e5fe4bbca8dd8cc52e42149f"
format_stamp: "Formatted at 2025-10-26 09:11:59 on dist-test-slave-w8k0"
I20251026 09:11:59.315850 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:59.320876 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:59.321105 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:59.323962 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.62:35803
I20251026 09:11:59.327267 9985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.62:35803 every 8 connection(s)
I20251026 09:11:59.327421 9986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:11:59.328536 9986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f: Bootstrap starting.
I20251026 09:11:59.328869 9986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f: Neither blocks nor log segments found. Creating new log.
I20251026 09:11:59.329454 9986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f: No bootstrap required, opened a new log
I20251026 09:11:59.329593 9986 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba52672e5fe4bbca8dd8cc52e42149f" member_type: VOTER }
I20251026 09:11:59.329650 9986 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:11:59.329665 9986 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba52672e5fe4bbca8dd8cc52e42149f, State: Initialized, Role: FOLLOWER
I20251026 09:11:59.329726 9986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba52672e5fe4bbca8dd8cc52e42149f" member_type: VOTER }
I20251026 09:11:59.329766 9986 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 09:11:59.329789 9986 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 09:11:59.329815 9986 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:11:59.330369 9986 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba52672e5fe4bbca8dd8cc52e42149f" member_type: VOTER }
I20251026 09:11:59.330438 9986 leader_election.cc:304] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ba52672e5fe4bbca8dd8cc52e42149f; no voters:
I20251026 09:11:59.330564 9986 leader_election.cc:290] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 09:11:59.330632 9989 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:11:59.330736 9986 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [sys.catalog]: configured and running, proceeding with master startup.
I20251026 09:11:59.330829 9989 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 1 LEADER]: Becoming Leader. State: Replica: 9ba52672e5fe4bbca8dd8cc52e42149f, State: Running, Role: LEADER
I20251026 09:11:59.330907 9989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba52672e5fe4bbca8dd8cc52e42149f" member_type: VOTER }
I20251026 09:11:59.331127 9990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ba52672e5fe4bbca8dd8cc52e42149f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba52672e5fe4bbca8dd8cc52e42149f" member_type: VOTER } }
I20251026 09:11:59.331184 9990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:59.331171 9991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ba52672e5fe4bbca8dd8cc52e42149f. Latest consensus state: current_term: 1 leader_uuid: "9ba52672e5fe4bbca8dd8cc52e42149f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba52672e5fe4bbca8dd8cc52e42149f" member_type: VOTER } }
I20251026 09:11:59.331250 9991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [sys.catalog]: This master's current role is: LEADER
I20251026 09:11:59.331458 9993 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 09:11:59.331660 9993 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 09:11:59.332312 9993 catalog_manager.cc:1357] Generated new cluster ID: 398d06c764bb46a4b6eb942fd042389b
I20251026 09:11:59.332319 3128 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 09:11:59.332371 9993 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 09:11:59.341722 9993 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 09:11:59.342207 9993 catalog_manager.cc:1514] Loading token signing keys...
I20251026 09:11:59.345729 9993 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f: Generated new TSK 0
I20251026 09:11:59.345939 9993 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 09:11:59.348093 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:59.349711 10008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:59.349897 10010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:59.349952 3128 server_base.cc:1047] running on GCE node
W20251026 09:11:59.350180 10007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:59.350334 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:59.350368 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:59.350394 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469919350392 us; error 0 us; skew 500 ppm
I20251026 09:11:59.351110 3128 webserver.cc:492] Webserver started at http://127.3.14.1:41759/ using document root <none> and password file <none>
I20251026 09:11:59.351194 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:59.351253 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:59.351310 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:59.351652 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root/instance:
uuid: "54f14d35dc7643e493354830387f763b"
format_stamp: "Formatted at 2025-10-26 09:11:59 on dist-test-slave-w8k0"
I20251026 09:11:59.352603 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:59.353107 10015 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.353256 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.353297 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "54f14d35dc7643e493354830387f763b"
format_stamp: "Formatted at 2025-10-26 09:11:59 on dist-test-slave-w8k0"
I20251026 09:11:59.353346 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:59.363679 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:59.363955 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:59.364128 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:59.364336 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:59.364365 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.364403 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:59.364428 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.367269 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:39349
I20251026 09:11:59.367307 10078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:39349 every 8 connection(s)
I20251026 09:11:59.367751 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:59.369259 10083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:59.369372 10084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:59.369289 10086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:59.369656 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:59.369792 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:59.369827 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:59.369846 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469919369846 us; error 0 us; skew 500 ppm
I20251026 09:11:59.370592 3128 webserver.cc:492] Webserver started at http://127.3.14.2:44837/ using document root <none> and password file <none>
I20251026 09:11:59.370705 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:59.370766 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:59.370826 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:59.371220 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root/instance:
uuid: "50176c38320e477db3a98358734890b2"
format_stamp: "Formatted at 2025-10-26 09:11:59 on dist-test-slave-w8k0"
I20251026 09:11:59.372198 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:11:59.372656 10091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.372800 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.372850 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root
uuid: "50176c38320e477db3a98358734890b2"
format_stamp: "Formatted at 2025-10-26 09:11:59 on dist-test-slave-w8k0"
I20251026 09:11:59.372900 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:59.377887 10079 heartbeater.cc:344] Connected to a master server at 127.3.14.62:35803
I20251026 09:11:59.377995 10079 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:59.378121 10079 heartbeater.cc:507] Master 127.3.14.62:35803 requested a full tablet report, sending...
I20251026 09:11:59.378388 9950 ts_manager.cc:194] Registered new tserver with Master: 54f14d35dc7643e493354830387f763b (127.3.14.1:39349)
I20251026 09:11:59.379213 9950 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53542
I20251026 09:11:59.389073 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:59.389324 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:59.389475 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:59.389676 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:59.389726 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.389751 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:59.389765 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.392684 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.2:42807
I20251026 09:11:59.392793 10154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.2:42807 every 8 connection(s)
I20251026 09:11:59.393071 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 09:11:59.394425 10159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:59.394510 10158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:11:59.394690 10161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:11:59.394800 3128 server_base.cc:1047] running on GCE node
I20251026 09:11:59.394937 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:11:59.394976 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:11:59.394997 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469919394998 us; error 0 us; skew 500 ppm
I20251026 09:11:59.395812 3128 webserver.cc:492] Webserver started at http://127.3.14.3:40317/ using document root <none> and password file <none>
I20251026 09:11:59.395923 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:11:59.395998 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:11:59.396060 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:11:59.396485 10155 heartbeater.cc:344] Connected to a master server at 127.3.14.62:35803
I20251026 09:11:59.396566 10155 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:59.396489 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root/instance:
uuid: "ac45b5ee4cdb400ba12e12617d0f800a"
format_stamp: "Formatted at 2025-10-26 09:11:59 on dist-test-slave-w8k0"
I20251026 09:11:59.396667 10155 heartbeater.cc:507] Master 127.3.14.62:35803 requested a full tablet report, sending...
I20251026 09:11:59.396926 9950 ts_manager.cc:194] Registered new tserver with Master: 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:11:59.397271 9950 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53558
I20251026 09:11:59.397745 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 09:11:59.398237 10166 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.398381 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.398432 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root
uuid: "ac45b5ee4cdb400ba12e12617d0f800a"
format_stamp: "Formatted at 2025-10-26 09:11:59 on dist-test-slave-w8k0"
I20251026 09:11:59.398492 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:11:59.412528 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:11:59.412799 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:11:59.413002 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:11:59.413220 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:11:59.413251 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.413290 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:11:59.413316 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:11:59.416445 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.3:34055
I20251026 09:11:59.416529 10229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.3:34055 every 8 connection(s)
I20251026 09:11:59.419910 10230 heartbeater.cc:344] Connected to a master server at 127.3.14.62:35803
I20251026 09:11:59.420004 10230 heartbeater.cc:461] Registering TS with master...
I20251026 09:11:59.420096 10230 heartbeater.cc:507] Master 127.3.14.62:35803 requested a full tablet report, sending...
I20251026 09:11:59.420347 9950 ts_manager.cc:194] Registered new tserver with Master: ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055)
I20251026 09:11:59.420666 9950 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53564
I20251026 09:11:59.421160 3128 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004487253s
I20251026 09:12:00.379868 10079 heartbeater.cc:499] Master 127.3.14.62:35803 was elected leader, sending a full tablet report...
I20251026 09:12:00.397895 10155 heartbeater.cc:499] Master 127.3.14.62:35803 was elected leader, sending a full tablet report...
I20251026 09:12:00.421226 10230 heartbeater.cc:499] Master 127.3.14.62:35803 was elected leader, sending a full tablet report...
I20251026 09:12:00.446720 3128 test_util.cc:276] Using random seed: 753142011
I20251026 09:12:00.452469 9950 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57294:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:12:00.452675 9950 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:12:00.458277 10191 tablet_service.cc:1505] Processing CreateTablet for tablet 4059517ece77454a810e8bc9f8a09a9f (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:12:00.458365 10192 tablet_service.cc:1505] Processing CreateTablet for tablet 36fa073fd46b48eca8f78b1e4f7926f1 (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:12:00.458276 10194 tablet_service.cc:1505] Processing CreateTablet for tablet f2f9f5c2c1e943809cb62f760ad811d6 (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:12:00.458334 10193 tablet_service.cc:1505] Processing CreateTablet for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:12:00.458520 10191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4059517ece77454a810e8bc9f8a09a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.458802 10119 tablet_service.cc:1505] Processing CreateTablet for tablet f2f9f5c2c1e943809cb62f760ad811d6 (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:12:00.458886 10119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f9f5c2c1e943809cb62f760ad811d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.459486 10116 tablet_service.cc:1505] Processing CreateTablet for tablet 4059517ece77454a810e8bc9f8a09a9f (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:12:00.459584 10116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4059517ece77454a810e8bc9f8a09a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.460147 10250 tablet_bootstrap.cc:492] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2: Bootstrap starting.
I20251026 09:12:00.460177 10043 tablet_service.cc:1505] Processing CreateTablet for tablet f2f9f5c2c1e943809cb62f760ad811d6 (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:12:00.460282 10043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f9f5c2c1e943809cb62f760ad811d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.460589 10250 tablet_bootstrap.cc:654] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.460897 10117 tablet_service.cc:1505] Processing CreateTablet for tablet 36fa073fd46b48eca8f78b1e4f7926f1 (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:12:00.461006 10117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fa073fd46b48eca8f78b1e4f7926f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.461163 10250 tablet_bootstrap.cc:492] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2: No bootstrap required, opened a new log
I20251026 09:12:00.461197 10250 ts_tablet_manager.cc:1403] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.461331 10250 raft_consensus.cc:359] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.461385 10250 raft_consensus.cc:385] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.461398 10250 raft_consensus.cc:740] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50176c38320e477db3a98358734890b2, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.461452 10250 consensus_queue.cc:260] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.461627 10250 ts_tablet_manager.cc:1434] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:12:00.461699 10250 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2: Bootstrap starting.
I20251026 09:12:00.461716 10042 tablet_service.cc:1505] Processing CreateTablet for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:12:00.461809 10042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2c3bdaf61b450e8b72ece96f440ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.462145 10250 tablet_bootstrap.cc:654] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.462334 10118 tablet_service.cc:1505] Processing CreateTablet for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:12:00.462421 10118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2c3bdaf61b450e8b72ece96f440ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.462735 10250 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2: No bootstrap required, opened a new log
I20251026 09:12:00.462769 10250 ts_tablet_manager.cc:1403] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.462925 10250 raft_consensus.cc:359] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.462977 10250 raft_consensus.cc:385] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.462991 10250 raft_consensus.cc:740] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50176c38320e477db3a98358734890b2, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.463030 10250 consensus_queue.cc:260] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.463074 10041 tablet_service.cc:1505] Processing CreateTablet for tablet 36fa073fd46b48eca8f78b1e4f7926f1 (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:12:00.463122 10250 ts_tablet_manager.cc:1434] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.463164 10041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fa073fd46b48eca8f78b1e4f7926f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.463654 10192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fa073fd46b48eca8f78b1e4f7926f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.464425 10040 tablet_service.cc:1505] Processing CreateTablet for tablet 4059517ece77454a810e8bc9f8a09a9f (DEFAULT_TABLE table=test-workload [id=afb0332ecfa94be5b89d94adb410d780]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:12:00.464493 10040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4059517ece77454a810e8bc9f8a09a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.465567 10193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2c3bdaf61b450e8b72ece96f440ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.466593 10194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2f9f5c2c1e943809cb62f760ad811d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.463177 10250 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2: Bootstrap starting.
I20251026 09:12:00.467850 10250 tablet_bootstrap.cc:654] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.468201 10251 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a: Bootstrap starting.
I20251026 09:12:00.468441 10250 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2: No bootstrap required, opened a new log
I20251026 09:12:00.468493 10250 ts_tablet_manager.cc:1403] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20251026 09:12:00.468595 10251 tablet_bootstrap.cc:654] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.468660 10250 raft_consensus.cc:359] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.468737 10250 raft_consensus.cc:385] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.468765 10250 raft_consensus.cc:740] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50176c38320e477db3a98358734890b2, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.468822 10250 consensus_queue.cc:260] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.468920 10250 ts_tablet_manager.cc:1434] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:12:00.468972 10254 tablet_bootstrap.cc:492] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:00.469012 10250 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2: Bootstrap starting.
I20251026 09:12:00.469209 10251 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a: No bootstrap required, opened a new log
I20251026 09:12:00.469255 10251 ts_tablet_manager.cc:1403] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.469403 10251 raft_consensus.cc:359] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.469470 10251 raft_consensus.cc:385] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.469480 10254 tablet_bootstrap.cc:654] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.469498 10251 raft_consensus.cc:740] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.469558 10250 tablet_bootstrap.cc:654] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.469578 10251 consensus_queue.cc:260] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.469801 10251 ts_tablet_manager.cc:1434] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.470026 10251 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a: Bootstrap starting.
I20251026 09:12:00.470212 10254 tablet_bootstrap.cc:492] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: No bootstrap required, opened a new log
I20251026 09:12:00.470256 10254 ts_tablet_manager.cc:1403] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.470403 10254 raft_consensus.cc:359] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.470465 10251 tablet_bootstrap.cc:654] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.470467 10254 raft_consensus.cc:385] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.470654 10254 raft_consensus.cc:740] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.470710 10254 consensus_queue.cc:260] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.470872 10254 ts_tablet_manager.cc:1434] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.470983 10250 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2: No bootstrap required, opened a new log
I20251026 09:12:00.471001 10254 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:00.471028 10250 ts_tablet_manager.cc:1403] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:12:00.471177 10250 raft_consensus.cc:359] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.471239 10250 raft_consensus.cc:385] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.471263 10250 raft_consensus.cc:740] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50176c38320e477db3a98358734890b2, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.471312 10250 consensus_queue.cc:260] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.471441 10254 tablet_bootstrap.cc:654] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.471442 10250 ts_tablet_manager.cc:1434] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.471665 10251 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a: No bootstrap required, opened a new log
I20251026 09:12:00.471731 10251 ts_tablet_manager.cc:1403] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 09:12:00.471885 10251 raft_consensus.cc:359] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.471948 10251 raft_consensus.cc:385] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.471971 10251 raft_consensus.cc:740] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.472018 10251 consensus_queue.cc:260] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.472095 10251 ts_tablet_manager.cc:1434] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.472177 10251 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a: Bootstrap starting.
I20251026 09:12:00.472292 10254 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: No bootstrap required, opened a new log
I20251026 09:12:00.472388 10254 ts_tablet_manager.cc:1403] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:00.472579 10251 tablet_bootstrap.cc:654] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.472577 10254 raft_consensus.cc:359] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.472683 10254 raft_consensus.cc:385] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.472705 10254 raft_consensus.cc:740] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.472761 10254 consensus_queue.cc:260] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.472851 10254 ts_tablet_manager.cc:1434] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.472911 10254 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:00.473304 10251 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a: No bootstrap required, opened a new log
I20251026 09:12:00.473331 10254 tablet_bootstrap.cc:654] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.473357 10251 ts_tablet_manager.cc:1403] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.473508 10251 raft_consensus.cc:359] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.473565 10251 raft_consensus.cc:385] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.473593 10251 raft_consensus.cc:740] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.473646 10251 consensus_queue.cc:260] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.473755 10251 ts_tablet_manager.cc:1434] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.473822 10251 tablet_bootstrap.cc:492] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a: Bootstrap starting.
I20251026 09:12:00.473930 10254 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: No bootstrap required, opened a new log
I20251026 09:12:00.473984 10254 ts_tablet_manager.cc:1403] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:00.474126 10254 raft_consensus.cc:359] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.474186 10254 raft_consensus.cc:385] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.474210 10254 raft_consensus.cc:740] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.474244 10251 tablet_bootstrap.cc:654] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.474287 10254 consensus_queue.cc:260] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.474395 10254 ts_tablet_manager.cc:1434] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.474457 10254 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:00.474905 10254 tablet_bootstrap.cc:654] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.474967 10251 tablet_bootstrap.cc:492] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a: No bootstrap required, opened a new log
I20251026 09:12:00.475034 10251 ts_tablet_manager.cc:1403] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.475203 10251 raft_consensus.cc:359] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.475261 10251 raft_consensus.cc:385] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.475289 10251 raft_consensus.cc:740] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.475365 10251 consensus_queue.cc:260] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.475502 10251 ts_tablet_manager.cc:1434] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:12:00.475533 10254 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: No bootstrap required, opened a new log
I20251026 09:12:00.475617 10254 ts_tablet_manager.cc:1403] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:00.475785 10254 raft_consensus.cc:359] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.475849 10254 raft_consensus.cc:385] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.475873 10254 raft_consensus.cc:740] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.475946 10254 consensus_queue.cc:260] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.476058 10254 ts_tablet_manager.cc:1434] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.569797 10253 raft_consensus.cc:493] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:12:00.569931 10253 raft_consensus.cc:515] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.570145 10253 leader_election.cc:290] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055)
I20251026 09:12:00.573956 10204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b2c3bdaf61b450e8b72ece96f440ee7" candidate_uuid: "50176c38320e477db3a98358734890b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" is_pre_election: true
I20251026 09:12:00.573948 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b2c3bdaf61b450e8b72ece96f440ee7" candidate_uuid: "50176c38320e477db3a98358734890b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b" is_pre_election: true
I20251026 09:12:00.574131 10204 raft_consensus.cc:2468] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50176c38320e477db3a98358734890b2 in term 0.
I20251026 09:12:00.574151 10053 raft_consensus.cc:2468] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50176c38320e477db3a98358734890b2 in term 0.
I20251026 09:12:00.574297 10095 leader_election.cc:304] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50176c38320e477db3a98358734890b2, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.574404 10253 raft_consensus.cc:2804] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:12:00.574472 10253 raft_consensus.cc:493] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:12:00.574507 10253 raft_consensus.cc:3060] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.575239 10253 raft_consensus.cc:515] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.575438 10253 leader_election.cc:290] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [CANDIDATE]: Term 1 election: Requested vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055)
I20251026 09:12:00.575603 10204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b2c3bdaf61b450e8b72ece96f440ee7" candidate_uuid: "50176c38320e477db3a98358734890b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac45b5ee4cdb400ba12e12617d0f800a"
I20251026 09:12:00.575662 10204 raft_consensus.cc:3060] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.576256 10204 raft_consensus.cc:2468] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50176c38320e477db3a98358734890b2 in term 1.
I20251026 09:12:00.576391 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b2c3bdaf61b450e8b72ece96f440ee7" candidate_uuid: "50176c38320e477db3a98358734890b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b"
I20251026 09:12:00.576448 10053 raft_consensus.cc:3060] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.576488 10095 leader_election.cc:304] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50176c38320e477db3a98358734890b2, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.576593 10253 raft_consensus.cc:2804] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:12:00.576761 10253 raft_consensus.cc:697] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Becoming Leader. State: Replica: 50176c38320e477db3a98358734890b2, State: Running, Role: LEADER
I20251026 09:12:00.576814 10253 consensus_queue.cc:237] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.577217 10053 raft_consensus.cc:2468] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50176c38320e477db3a98358734890b2 in term 1.
I20251026 09:12:00.577471 9949 catalog_manager.cc:5649] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50176c38320e477db3a98358734890b2 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: UNKNOWN } } }
I20251026 09:12:00.615037 10258 raft_consensus.cc:493] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:12:00.615168 10258 raft_consensus.cc:515] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.615465 10258 leader_election.cc:290] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50176c38320e477db3a98358734890b2 (127.3.14.2:42807), 54f14d35dc7643e493354830387f763b (127.3.14.1:39349)
I20251026 09:12:00.619318 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2f9f5c2c1e943809cb62f760ad811d6" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2" is_pre_election: true
I20251026 09:12:00.619300 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2f9f5c2c1e943809cb62f760ad811d6" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b" is_pre_election: true
I20251026 09:12:00.619488 10129 raft_consensus.cc:2468] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.619496 10053 raft_consensus.cc:2468] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.619714 10170 leader_election.cc:304] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50176c38320e477db3a98358734890b2, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.619833 10258 raft_consensus.cc:2804] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:12:00.619877 10258 raft_consensus.cc:493] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:12:00.619897 10258 raft_consensus.cc:3060] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.620618 10258 raft_consensus.cc:515] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.620776 10258 leader_election.cc:290] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Requested vote from peers 50176c38320e477db3a98358734890b2 (127.3.14.2:42807), 54f14d35dc7643e493354830387f763b (127.3.14.1:39349)
I20251026 09:12:00.620952 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2f9f5c2c1e943809cb62f760ad811d6" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b"
I20251026 09:12:00.620952 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2f9f5c2c1e943809cb62f760ad811d6" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2"
I20251026 09:12:00.621054 10053 raft_consensus.cc:3060] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.621061 10129 raft_consensus.cc:3060] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.621789 10129 raft_consensus.cc:2468] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.621789 10053 raft_consensus.cc:2468] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.622077 10167 leader_election.cc:304] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f14d35dc7643e493354830387f763b, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.622186 10258 raft_consensus.cc:2804] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:12:00.622337 10258 raft_consensus.cc:697] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Becoming Leader. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Running, Role: LEADER
I20251026 09:12:00.622395 10258 consensus_queue.cc:237] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.623057 9949 catalog_manager.cc:5649] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a reported cstate change: term changed from 0 to 1, leader changed from <none> to ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3). New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: HEALTHY } } }
I20251026 09:12:00.634334 10253 raft_consensus.cc:493] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:12:00.634459 10253 raft_consensus.cc:515] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.634654 10253 leader_election.cc:290] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055)
I20251026 09:12:00.634855 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36fa073fd46b48eca8f78b1e4f7926f1" candidate_uuid: "50176c38320e477db3a98358734890b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b" is_pre_election: true
I20251026 09:12:00.634855 10204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36fa073fd46b48eca8f78b1e4f7926f1" candidate_uuid: "50176c38320e477db3a98358734890b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" is_pre_election: true
I20251026 09:12:00.634965 10053 raft_consensus.cc:2468] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50176c38320e477db3a98358734890b2 in term 0.
I20251026 09:12:00.634967 10204 raft_consensus.cc:2468] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50176c38320e477db3a98358734890b2 in term 0.
I20251026 09:12:00.635147 10092 leader_election.cc:304] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50176c38320e477db3a98358734890b2, 54f14d35dc7643e493354830387f763b; no voters:
I20251026 09:12:00.635263 10253 raft_consensus.cc:2804] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:12:00.635299 10253 raft_consensus.cc:493] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:12:00.635361 10253 raft_consensus.cc:3060] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.636050 10253 raft_consensus.cc:515] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.636196 10253 leader_election.cc:290] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [CANDIDATE]: Term 1 election: Requested vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055)
I20251026 09:12:00.636415 10204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36fa073fd46b48eca8f78b1e4f7926f1" candidate_uuid: "50176c38320e477db3a98358734890b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac45b5ee4cdb400ba12e12617d0f800a"
I20251026 09:12:00.636478 10204 raft_consensus.cc:3060] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.637115 10204 raft_consensus.cc:2468] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50176c38320e477db3a98358734890b2 in term 1.
I20251026 09:12:00.637219 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36fa073fd46b48eca8f78b1e4f7926f1" candidate_uuid: "50176c38320e477db3a98358734890b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b"
I20251026 09:12:00.637288 10053 raft_consensus.cc:3060] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.637312 10095 leader_election.cc:304] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50176c38320e477db3a98358734890b2, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.637451 10253 raft_consensus.cc:2804] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:12:00.637511 10253 raft_consensus.cc:697] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Becoming Leader. State: Replica: 50176c38320e477db3a98358734890b2, State: Running, Role: LEADER
I20251026 09:12:00.637560 10253 consensus_queue.cc:237] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.638026 10053 raft_consensus.cc:2468] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50176c38320e477db3a98358734890b2 in term 1.
I20251026 09:12:00.638228 9949 catalog_manager.cc:5649] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50176c38320e477db3a98358734890b2 (127.3.14.2). New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: UNKNOWN } } }
I20251026 09:12:00.639010 10258 raft_consensus.cc:493] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:12:00.639070 10258 raft_consensus.cc:515] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.639199 10258 leader_election.cc:290] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:00.639381 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4059517ece77454a810e8bc9f8a09a9f" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2" is_pre_election: true
I20251026 09:12:00.639386 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4059517ece77454a810e8bc9f8a09a9f" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b" is_pre_election: true
I20251026 09:12:00.639453 10129 raft_consensus.cc:2468] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.639474 10053 raft_consensus.cc:2468] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.639569 10170 leader_election.cc:304] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50176c38320e477db3a98358734890b2, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.639675 10258 raft_consensus.cc:2804] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:12:00.639742 10258 raft_consensus.cc:493] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:12:00.639763 10258 raft_consensus.cc:3060] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.640321 10258 raft_consensus.cc:515] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.640432 10258 leader_election.cc:290] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Requested vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:00.640594 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4059517ece77454a810e8bc9f8a09a9f" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b"
I20251026 09:12:00.640633 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4059517ece77454a810e8bc9f8a09a9f" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2"
I20251026 09:12:00.640656 10053 raft_consensus.cc:3060] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.640676 10129 raft_consensus.cc:3060] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.641225 10053 raft_consensus.cc:2468] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.641239 10129 raft_consensus.cc:2468] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.641412 10170 leader_election.cc:304] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50176c38320e477db3a98358734890b2, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.641502 10258 raft_consensus.cc:2804] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:12:00.641553 10258 raft_consensus.cc:697] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Becoming Leader. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Running, Role: LEADER
I20251026 09:12:00.641588 10258 consensus_queue.cc:237] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.642184 9949 catalog_manager.cc:5649] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a reported cstate change: term changed from 0 to 1, leader changed from <none> to ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3). New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: HEALTHY } } }
I20251026 09:12:00.681777 3128 test_util.cc:276] Using random seed: 753377062
I20251026 09:12:00.687974 9949 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57302:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 09:12:00.688174 9949 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 09:12:00.689939 10040 tablet_service.cc:1505] Processing CreateTablet for tablet ddfa546f8bd8478380c63e2803d0e153 (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:12:00.690035 10117 tablet_service.cc:1505] Processing CreateTablet for tablet 8b3912b615dd4121a79577c49631f14a (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:12:00.690109 10043 tablet_service.cc:1505] Processing CreateTablet for tablet 213642ddd7c749b3baee0f01cd14de97 (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:12:00.690137 10040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddfa546f8bd8478380c63e2803d0e153. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.690187 10043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 213642ddd7c749b3baee0f01cd14de97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.690161 10118 tablet_service.cc:1505] Processing CreateTablet for tablet ddfa546f8bd8478380c63e2803d0e153 (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:12:00.690230 10116 tablet_service.cc:1505] Processing CreateTablet for tablet 95cc5ade38624205bad56f670c1ea47d (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:12:00.689944 10042 tablet_service.cc:1505] Processing CreateTablet for tablet 95cc5ade38624205bad56f670c1ea47d (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:12:00.690174 10117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3912b615dd4121a79577c49631f14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.690338 10042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95cc5ade38624205bad56f670c1ea47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.690356 10194 tablet_service.cc:1505] Processing CreateTablet for tablet ddfa546f8bd8478380c63e2803d0e153 (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 09:12:00.690454 10194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddfa546f8bd8478380c63e2803d0e153. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.691706 10118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddfa546f8bd8478380c63e2803d0e153. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.691727 10251 tablet_bootstrap.cc:492] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: Bootstrap starting.
I20251026 09:12:00.691779 10192 tablet_service.cc:1505] Processing CreateTablet for tablet 95cc5ade38624205bad56f670c1ea47d (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 09:12:00.691869 10192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95cc5ade38624205bad56f670c1ea47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.692222 10251 tablet_bootstrap.cc:654] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.692600 10250 tablet_bootstrap.cc:492] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: Bootstrap starting.
I20251026 09:12:00.693024 10254 tablet_bootstrap.cc:492] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:00.693051 10250 tablet_bootstrap.cc:654] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.691742 10193 tablet_service.cc:1505] Processing CreateTablet for tablet 8b3912b615dd4121a79577c49631f14a (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:12:00.693410 10193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3912b615dd4121a79577c49631f14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.693538 10254 tablet_bootstrap.cc:654] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.693681 10250 tablet_bootstrap.cc:492] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: No bootstrap required, opened a new log
I20251026 09:12:00.693742 10250 ts_tablet_manager.cc:1403] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.690239 10041 tablet_service.cc:1505] Processing CreateTablet for tablet 8b3912b615dd4121a79577c49631f14a (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 09:12:00.693928 10250 raft_consensus.cc:359] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.693997 10041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3912b615dd4121a79577c49631f14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.694012 10250 raft_consensus.cc:385] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.694037 10250 raft_consensus.cc:740] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50176c38320e477db3a98358734890b2, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.694092 10250 consensus_queue.cc:260] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.694190 10250 ts_tablet_manager.cc:1434] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:12:00.694248 10250 tablet_bootstrap.cc:492] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: Bootstrap starting.
I20251026 09:12:00.694643 10250 tablet_bootstrap.cc:654] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.694792 10118 tablet_service.cc:1505] Processing CreateTablet for tablet 213642ddd7c749b3baee0f01cd14de97 (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:12:00.694870 10118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 213642ddd7c749b3baee0f01cd14de97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.695307 10250 tablet_bootstrap.cc:492] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: No bootstrap required, opened a new log
I20251026 09:12:00.695387 10250 ts_tablet_manager.cc:1403] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.695499 10116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95cc5ade38624205bad56f670c1ea47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.695549 10250 raft_consensus.cc:359] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.695624 10250 raft_consensus.cc:385] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.695653 10250 raft_consensus.cc:740] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50176c38320e477db3a98358734890b2, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.695708 10250 consensus_queue.cc:260] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.695803 10250 ts_tablet_manager.cc:1434] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.695861 10250 tablet_bootstrap.cc:492] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: Bootstrap starting.
I20251026 09:12:00.696628 10193 tablet_service.cc:1505] Processing CreateTablet for tablet 213642ddd7c749b3baee0f01cd14de97 (DEFAULT_TABLE table=dugtrio [id=c51f7950685c44239f6561c30028ccc2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 09:12:00.696708 10250 tablet_bootstrap.cc:654] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.696732 10193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 213642ddd7c749b3baee0f01cd14de97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:00.697078 10254 tablet_bootstrap.cc:492] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: No bootstrap required, opened a new log
I20251026 09:12:00.697139 10254 ts_tablet_manager.cc:1403] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20251026 09:12:00.697266 10254 raft_consensus.cc:359] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.697309 10254 raft_consensus.cc:385] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.697337 10254 raft_consensus.cc:740] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.697310 10251 tablet_bootstrap.cc:492] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: No bootstrap required, opened a new log
I20251026 09:12:00.697405 10251 ts_tablet_manager.cc:1403] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20251026 09:12:00.697389 10254 consensus_queue.cc:260] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.697510 10254 ts_tablet_manager.cc:1434] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.697566 10254 tablet_bootstrap.cc:492] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:00.697551 10251 raft_consensus.cc:359] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.697598 10251 raft_consensus.cc:385] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.697618 10251 raft_consensus.cc:740] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.697675 10251 consensus_queue.cc:260] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.697774 10251 ts_tablet_manager.cc:1434] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.697832 10251 tablet_bootstrap.cc:492] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: Bootstrap starting.
I20251026 09:12:00.698009 10254 tablet_bootstrap.cc:654] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.698259 10251 tablet_bootstrap.cc:654] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.698611 10254 tablet_bootstrap.cc:492] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: No bootstrap required, opened a new log
I20251026 09:12:00.698673 10254 ts_tablet_manager.cc:1403] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:00.698776 10250 tablet_bootstrap.cc:492] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: No bootstrap required, opened a new log
I20251026 09:12:00.698827 10250 ts_tablet_manager.cc:1403] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251026 09:12:00.698860 10254 raft_consensus.cc:359] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.698925 10254 raft_consensus.cc:385] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.698964 10254 raft_consensus.cc:740] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.699002 10250 raft_consensus.cc:359] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.699070 10250 raft_consensus.cc:385] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.699059 10251 tablet_bootstrap.cc:492] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: No bootstrap required, opened a new log
I20251026 09:12:00.699117 10250 raft_consensus.cc:740] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50176c38320e477db3a98358734890b2, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.699146 10251 ts_tablet_manager.cc:1403] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:00.699321 10251 raft_consensus.cc:359] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.699414 10251 raft_consensus.cc:385] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.699440 10251 raft_consensus.cc:740] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.699492 10251 consensus_queue.cc:260] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.699604 10251 ts_tablet_manager.cc:1434] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.699674 10251 tablet_bootstrap.cc:492] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: Bootstrap starting.
I20251026 09:12:00.700081 10251 tablet_bootstrap.cc:654] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.699153 10250 consensus_queue.cc:260] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.699013 10254 consensus_queue.cc:260] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.700528 10250 ts_tablet_manager.cc:1434] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:12:00.700580 10254 ts_tablet_manager.cc:1434] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:12:00.700601 10250 tablet_bootstrap.cc:492] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: Bootstrap starting.
I20251026 09:12:00.700646 10251 tablet_bootstrap.cc:492] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: No bootstrap required, opened a new log
I20251026 09:12:00.700708 10251 ts_tablet_manager.cc:1403] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:00.700858 10251 raft_consensus.cc:359] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.700917 10251 raft_consensus.cc:385] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.700951 10251 raft_consensus.cc:740] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.700994 10251 consensus_queue.cc:260] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.701061 10250 tablet_bootstrap.cc:654] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.701092 10251 ts_tablet_manager.cc:1434] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.701160 10251 tablet_bootstrap.cc:492] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: Bootstrap starting.
I20251026 09:12:00.701562 10251 tablet_bootstrap.cc:654] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.701663 10250 tablet_bootstrap.cc:492] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: No bootstrap required, opened a new log
I20251026 09:12:00.701699 10250 ts_tablet_manager.cc:1403] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:00.701817 10250 raft_consensus.cc:359] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.701851 10250 raft_consensus.cc:385] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.701866 10250 raft_consensus.cc:740] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50176c38320e477db3a98358734890b2, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.700646 10254 tablet_bootstrap.cc:492] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:00.701906 10250 consensus_queue.cc:260] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.702016 10250 ts_tablet_manager.cc:1434] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.702157 10251 tablet_bootstrap.cc:492] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: No bootstrap required, opened a new log
I20251026 09:12:00.702184 10251 ts_tablet_manager.cc:1403] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:12:00.702288 10251 raft_consensus.cc:359] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.702320 10251 raft_consensus.cc:385] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.702333 10251 raft_consensus.cc:740] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.702358 10254 tablet_bootstrap.cc:654] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.702385 10251 consensus_queue.cc:260] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.702489 10251 ts_tablet_manager.cc:1434] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.703182 10254 tablet_bootstrap.cc:492] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: No bootstrap required, opened a new log
I20251026 09:12:00.703234 10254 ts_tablet_manager.cc:1403] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251026 09:12:00.703405 10254 raft_consensus.cc:359] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.703483 10254 raft_consensus.cc:385] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.703512 10254 raft_consensus.cc:740] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.703562 10254 consensus_queue.cc:260] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.703652 10254 ts_tablet_manager.cc:1434] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.703729 10254 tablet_bootstrap.cc:492] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:00.704151 10254 tablet_bootstrap.cc:654] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Neither blocks nor log segments found. Creating new log.
I20251026 09:12:00.704728 10254 tablet_bootstrap.cc:492] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: No bootstrap required, opened a new log
I20251026 09:12:00.704775 10254 ts_tablet_manager.cc:1403] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:00.704907 10254 raft_consensus.cc:359] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.704960 10254 raft_consensus.cc:385] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 09:12:00.704986 10254 raft_consensus.cc:740] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:00.705049 10254 consensus_queue.cc:260] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.705133 10254 ts_tablet_manager.cc:1434] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:00.734254 10258 raft_consensus.cc:493] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:12:00.734385 10258 raft_consensus.cc:515] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.734562 10258 leader_election.cc:290] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:00.734745 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddfa546f8bd8478380c63e2803d0e153" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2" is_pre_election: true
I20251026 09:12:00.734722 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddfa546f8bd8478380c63e2803d0e153" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b" is_pre_election: true
I20251026 09:12:00.734848 10129 raft_consensus.cc:2468] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.734853 10053 raft_consensus.cc:2468] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.735001 10167 leader_election.cc:304] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f14d35dc7643e493354830387f763b, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.735092 10258 raft_consensus.cc:2804] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:12:00.735137 10258 raft_consensus.cc:493] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:12:00.735164 10258 raft_consensus.cc:3060] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.735867 10258 raft_consensus.cc:515] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.736001 10258 leader_election.cc:290] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Requested vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:00.736183 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddfa546f8bd8478380c63e2803d0e153" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b"
I20251026 09:12:00.736183 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddfa546f8bd8478380c63e2803d0e153" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2"
I20251026 09:12:00.736277 10053 raft_consensus.cc:3060] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.736290 10129 raft_consensus.cc:3060] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.736975 10053 raft_consensus.cc:2468] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.736976 10129 raft_consensus.cc:2468] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.737201 10167 leader_election.cc:304] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f14d35dc7643e493354830387f763b, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.737318 10258 raft_consensus.cc:2804] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:12:00.737392 10258 raft_consensus.cc:697] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Becoming Leader. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Running, Role: LEADER
I20251026 09:12:00.737447 10258 consensus_queue.cc:237] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.738121 9949 catalog_manager.cc:5649] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a reported cstate change: term changed from 0 to 1, leader changed from <none> to ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3). New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: HEALTHY } } }
I20251026 09:12:00.765820 10258 raft_consensus.cc:493] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:12:00.765949 10258 raft_consensus.cc:515] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.766094 10258 leader_election.cc:290] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:00.766353 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95cc5ade38624205bad56f670c1ea47d" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2" is_pre_election: true
I20251026 09:12:00.766363 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95cc5ade38624205bad56f670c1ea47d" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b" is_pre_election: true
I20251026 09:12:00.766443 10129 raft_consensus.cc:2468] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.766477 10053 raft_consensus.cc:2468] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.766610 10170 leader_election.cc:304] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50176c38320e477db3a98358734890b2, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.766750 10258 raft_consensus.cc:2804] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:12:00.766796 10258 raft_consensus.cc:493] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:12:00.766813 10258 raft_consensus.cc:3060] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.767524 10258 raft_consensus.cc:515] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.767666 10258 leader_election.cc:290] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Requested vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:00.767838 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95cc5ade38624205bad56f670c1ea47d" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2"
I20251026 09:12:00.767838 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95cc5ade38624205bad56f670c1ea47d" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b"
I20251026 09:12:00.767920 10129 raft_consensus.cc:3060] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.767938 10053 raft_consensus.cc:3060] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.768513 10053 raft_consensus.cc:2468] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.768530 10129 raft_consensus.cc:2468] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.768702 10167 leader_election.cc:304] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f14d35dc7643e493354830387f763b, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.768802 10258 raft_consensus.cc:2804] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:12:00.768859 10258 raft_consensus.cc:697] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Becoming Leader. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Running, Role: LEADER
I20251026 09:12:00.768920 10258 consensus_queue.cc:237] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.769556 9949 catalog_manager.cc:5649] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a reported cstate change: term changed from 0 to 1, leader changed from <none> to ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3). New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: HEALTHY } } }
I20251026 09:12:00.825340 10262 raft_consensus.cc:493] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:12:00.825461 10262 raft_consensus.cc:515] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.825692 10262 leader_election.cc:290] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50176c38320e477db3a98358734890b2 (127.3.14.2:42807), ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055)
I20251026 09:12:00.828727 10204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b3912b615dd4121a79577c49631f14a" candidate_uuid: "54f14d35dc7643e493354830387f763b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" is_pre_election: true
I20251026 09:12:00.828686 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b3912b615dd4121a79577c49631f14a" candidate_uuid: "54f14d35dc7643e493354830387f763b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2" is_pre_election: true
I20251026 09:12:00.828852 10204 raft_consensus.cc:2468] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54f14d35dc7643e493354830387f763b in term 0.
I20251026 09:12:00.828860 10129 raft_consensus.cc:2468] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54f14d35dc7643e493354830387f763b in term 0.
I20251026 09:12:00.829034 10019 leader_election.cc:304] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f14d35dc7643e493354830387f763b, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.829159 10262 raft_consensus.cc:2804] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:12:00.829234 10262 raft_consensus.cc:493] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:12:00.829262 10262 raft_consensus.cc:3060] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.829969 10262 raft_consensus.cc:515] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.830133 10262 leader_election.cc:290] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [CANDIDATE]: Term 1 election: Requested vote from peers 50176c38320e477db3a98358734890b2 (127.3.14.2:42807), ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055)
I20251026 09:12:00.830302 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b3912b615dd4121a79577c49631f14a" candidate_uuid: "54f14d35dc7643e493354830387f763b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2"
I20251026 09:12:00.830369 10129 raft_consensus.cc:3060] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.830343 10204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b3912b615dd4121a79577c49631f14a" candidate_uuid: "54f14d35dc7643e493354830387f763b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac45b5ee4cdb400ba12e12617d0f800a"
I20251026 09:12:00.830407 10204 raft_consensus.cc:3060] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.830986 10129 raft_consensus.cc:2468] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f14d35dc7643e493354830387f763b in term 1.
I20251026 09:12:00.830989 10204 raft_consensus.cc:2468] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f14d35dc7643e493354830387f763b in term 1.
I20251026 09:12:00.831189 10019 leader_election.cc:304] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f14d35dc7643e493354830387f763b, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.831295 10262 raft_consensus.cc:2804] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:12:00.831468 10262 raft_consensus.cc:697] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 LEADER]: Becoming Leader. State: Replica: 54f14d35dc7643e493354830387f763b, State: Running, Role: LEADER
I20251026 09:12:00.831519 10262 consensus_queue.cc:237] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.832172 9949 catalog_manager.cc:5649] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b reported cstate change: term changed from 0 to 1, leader changed from <none> to 54f14d35dc7643e493354830387f763b (127.3.14.1). New cstate: current_term: 1 leader_uuid: "54f14d35dc7643e493354830387f763b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: UNKNOWN } } }
I20251026 09:12:00.856395 10258 raft_consensus.cc:493] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 09:12:00.856518 10258 raft_consensus.cc:515] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.856734 10258 leader_election.cc:290] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:00.856963 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "213642ddd7c749b3baee0f01cd14de97" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b" is_pre_election: true
I20251026 09:12:00.856966 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "213642ddd7c749b3baee0f01cd14de97" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2" is_pre_election: true
I20251026 09:12:00.857074 10053 raft_consensus.cc:2468] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.857079 10129 raft_consensus.cc:2468] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 0.
I20251026 09:12:00.857239 10167 leader_election.cc:304] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f14d35dc7643e493354830387f763b, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.857352 10258 raft_consensus.cc:2804] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 09:12:00.857402 10258 raft_consensus.cc:493] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 09:12:00.857421 10258 raft_consensus.cc:3060] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.858093 10258 raft_consensus.cc:515] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.858237 10258 leader_election.cc:290] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Requested vote from peers 54f14d35dc7643e493354830387f763b (127.3.14.1:39349), 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:00.858402 10053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "213642ddd7c749b3baee0f01cd14de97" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f14d35dc7643e493354830387f763b"
I20251026 09:12:00.858443 10129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "213642ddd7c749b3baee0f01cd14de97" candidate_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50176c38320e477db3a98358734890b2"
I20251026 09:12:00.858474 10053 raft_consensus.cc:3060] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.858498 10129 raft_consensus.cc:3060] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 09:12:00.859064 10053 raft_consensus.cc:2468] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.859074 10129 raft_consensus.cc:2468] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac45b5ee4cdb400ba12e12617d0f800a in term 1.
I20251026 09:12:00.859277 10167 leader_election.cc:304] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f14d35dc7643e493354830387f763b, ac45b5ee4cdb400ba12e12617d0f800a; no voters:
I20251026 09:12:00.859392 10258 raft_consensus.cc:2804] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 09:12:00.859460 10258 raft_consensus.cc:697] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Becoming Leader. State: Replica: ac45b5ee4cdb400ba12e12617d0f800a, State: Running, Role: LEADER
I20251026 09:12:00.859516 10258 consensus_queue.cc:237] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:00.860157 9949 catalog_manager.cc:5649] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a reported cstate change: term changed from 0 to 1, leader changed from <none> to ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3). New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: HEALTHY } } }
I20251026 09:12:00.880180 3128 tablet_server.cc:178] TabletServer@127.3.14.1:0 shutting down...
I20251026 09:12:00.883049 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:12:00.883255 3128 tablet_replica.cc:333] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: stopping tablet replica
I20251026 09:12:00.883359 3128 raft_consensus.cc:2243] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:00.883417 3128 raft_consensus.cc:2272] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:00.883633 3128 tablet_replica.cc:333] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: stopping tablet replica
I20251026 09:12:00.883675 3128 raft_consensus.cc:2243] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:00.883697 3128 raft_consensus.cc:2272] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:00.883847 3128 tablet_replica.cc:333] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: stopping tablet replica
I20251026 09:12:00.883889 3128 raft_consensus.cc:2243] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:12:00.883926 3128 pending_rounds.cc:70] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Trying to abort 1 pending ops.
I20251026 09:12:00.883946 3128 pending_rounds.cc:77] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214980795726778368 op_type: NO_OP noop_request { }
I20251026 09:12:00.884001 3128 raft_consensus.cc:2889] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 09:12:00.884023 3128 raft_consensus.cc:2272] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:00.884255 3128 tablet_replica.cc:333] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: stopping tablet replica
I20251026 09:12:00.884302 3128 raft_consensus.cc:2243] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:00.884325 3128 raft_consensus.cc:2272] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:00.884455 3128 tablet_replica.cc:333] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: stopping tablet replica
I20251026 09:12:00.884487 3128 raft_consensus.cc:2243] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:00.884505 3128 raft_consensus.cc:2272] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:00.884635 3128 tablet_replica.cc:333] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: stopping tablet replica
I20251026 09:12:00.884667 3128 raft_consensus.cc:2243] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:00.884685 3128 raft_consensus.cc:2272] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:00.884864 3128 tablet_replica.cc:333] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: stopping tablet replica
I20251026 09:12:00.884910 3128 raft_consensus.cc:2243] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:00.884931 3128 raft_consensus.cc:2272] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:00.885146 3128 tablet_replica.cc:333] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: stopping tablet replica
I20251026 09:12:00.885185 3128 raft_consensus.cc:2243] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:00.885205 3128 raft_consensus.cc:2272] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:00.887470 3128 tablet_server.cc:195] TabletServer@127.3.14.1:0 shutdown complete.
I20251026 09:12:00.998261 10253 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:12:01.000265 10092 consensus_peers.cc:597] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 -> Peer 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): Couldn't send request to peer 54f14d35dc7643e493354830387f763b. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:39349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:01.116979 10258 consensus_queue.cc:1048] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:12:01.118726 10167 consensus_peers.cc:597] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a -> Peer 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): Couldn't send request to peer 54f14d35dc7643e493354830387f763b. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:39349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:01.137144 10308 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:12:01.138999 10167 consensus_peers.cc:597] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a -> Peer 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): Couldn't send request to peer 54f14d35dc7643e493354830387f763b. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:39349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251026 09:12:01.160537 10167 consensus_peers.cc:597] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a -> Peer 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): Couldn't send request to peer 54f14d35dc7643e493354830387f763b. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:39349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251026 09:12:01.246598 10092 consensus_peers.cc:597] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 -> Peer 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): Couldn't send request to peer 54f14d35dc7643e493354830387f763b. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:39349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:01.254528 10304 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:12:01.285356 10167 consensus_peers.cc:597] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a -> Peer 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): Couldn't send request to peer 54f14d35dc7643e493354830387f763b. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:39349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:01.349025 10308 consensus_queue.cc:1048] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:12:01.372459 10258 consensus_queue.cc:1048] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:12:01.404362 10167 consensus_peers.cc:597] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a -> Peer 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): Couldn't send request to peer 54f14d35dc7643e493354830387f763b. Status: Network error: Client connection negotiation failed: client connection to 127.3.14.1:39349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:01.454203 10258 consensus_queue.cc:1048] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:12:01.685052 10253 consensus_queue.cc:579] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Leader has been unable to successfully communicate with peer 54f14d35dc7643e493354830387f763b for more than 1 seconds (1.047s)
W20251026 09:12:01.685746 9949 catalog_manager.cc:5327] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 36fa073fd46b48eca8f78b1e4f7926f1 with cas_config_opid_index -1: no extra replica candidate found for tablet 36fa073fd46b48eca8f78b1e4f7926f1 (table test-workload [id=afb0332ecfa94be5b89d94adb410d780]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20251026 09:12:01.755031 10258 consensus_queue.cc:579] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Leader has been unable to successfully communicate with peer 54f14d35dc7643e493354830387f763b for more than 1 seconds (1.113s)
I20251026 09:12:01.820533 10258 consensus_queue.cc:579] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Leader has been unable to successfully communicate with peer 54f14d35dc7643e493354830387f763b for more than 1 seconds (1.083s)
I20251026 09:12:01.892688 9949 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57294:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20251026 09:12:01.892813 9949 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57294:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20251026 09:12:01.894037 9949 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f: Sending DeleteTablet for 3 replicas of tablet ddfa546f8bd8478380c63e2803d0e153
I20251026 09:12:01.894186 9949 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f: Sending DeleteTablet for 3 replicas of tablet 8b3912b615dd4121a79577c49631f14a
I20251026 09:12:01.894244 9949 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f: Sending DeleteTablet for 3 replicas of tablet 95cc5ade38624205bad56f670c1ea47d
I20251026 09:12:01.894294 9949 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f: Sending DeleteTablet for 3 replicas of tablet 213642ddd7c749b3baee0f01cd14de97
I20251026 09:12:01.894512 10191 tablet_service.cc:1552] Processing DeleteTablet for tablet 213642ddd7c749b3baee0f01cd14de97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:51034
I20251026 09:12:01.894556 10116 tablet_service.cc:1552] Processing DeleteTablet for tablet ddfa546f8bd8478380c63e2803d0e153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:54650
I20251026 09:12:01.894532 10193 tablet_service.cc:1552] Processing DeleteTablet for tablet ddfa546f8bd8478380c63e2803d0e153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:51034
I20251026 09:12:01.894694 10119 tablet_service.cc:1552] Processing DeleteTablet for tablet 213642ddd7c749b3baee0f01cd14de97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:54650
I20251026 09:12:01.894760 10117 tablet_service.cc:1552] Processing DeleteTablet for tablet 95cc5ade38624205bad56f670c1ea47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:54650
I20251026 09:12:01.894793 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:12:01.894820 10118 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b3912b615dd4121a79577c49631f14a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:54650
I20251026 09:12:01.894495 10194 tablet_service.cc:1552] Processing DeleteTablet for tablet 95cc5ade38624205bad56f670c1ea47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:51034
I20251026 09:12:01.894867 10323 tablet_replica.cc:333] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: stopping tablet replica
I20251026 09:12:01.894922 10322 tablet_replica.cc:333] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: stopping tablet replica
I20251026 09:12:01.894485 10192 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b3912b615dd4121a79577c49631f14a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:51034
I20251026 09:12:01.894980 10322 raft_consensus.cc:2243] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:12:01.895001 10323 raft_consensus.cc:2243] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:01.895042 10323 raft_consensus.cc:2272] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.895068 10322 raft_consensus.cc:2272] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.895413 10322 ts_tablet_manager.cc:1916] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 09:12:01.895612 9934 catalog_manager.cc:4712] TS 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b3912b615dd4121a79577c49631f14a: Network error: Client connection negotiation failed: client connection to 127.3.14.1:39349: connect: Connection refused (error 111)
I20251026 09:12:01.895503 10323 ts_tablet_manager.cc:1916] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 09:12:01.896623 10325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:12:01.896764 10326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:12:01.896967 10328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:12:01.897346 3128 server_base.cc:1047] running on GCE node
W20251026 09:12:01.897492 10170 consensus_peers.cc:597] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a -> Peer 50176c38320e477db3a98358734890b2 (127.3.14.2:42807): Couldn't send request to peer 50176c38320e477db3a98358734890b2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:01.897559 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:12:01.897595 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:12:01.897619 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469921897618 us; error 0 us; skew 500 ppm
I20251026 09:12:01.897711 10322 ts_tablet_manager.cc:1929] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 09:12:01.897758 10322 log.cc:1199] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/213642ddd7c749b3baee0f01cd14de97
I20251026 09:12:01.897846 10323 ts_tablet_manager.cc:1929] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 09:12:01.897887 10323 log.cc:1199] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/ddfa546f8bd8478380c63e2803d0e153
I20251026 09:12:01.897980 10322 ts_tablet_manager.cc:1950] T 213642ddd7c749b3baee0f01cd14de97 P ac45b5ee4cdb400ba12e12617d0f800a: Deleting consensus metadata
I20251026 09:12:01.898105 10323 ts_tablet_manager.cc:1950] T ddfa546f8bd8478380c63e2803d0e153 P 50176c38320e477db3a98358734890b2: Deleting consensus metadata
I20251026 09:12:01.898408 10323 tablet_replica.cc:333] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: stopping tablet replica
I20251026 09:12:01.898476 10323 raft_consensus.cc:2243] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:01.898519 10323 raft_consensus.cc:2272] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.898507 9937 catalog_manager.cc:4985] TS ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055): tablet 213642ddd7c749b3baee0f01cd14de97 (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.898511 10322 tablet_replica.cc:333] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: stopping tablet replica
I20251026 09:12:01.898600 9937 catalog_manager.cc:4985] TS 50176c38320e477db3a98358734890b2 (127.3.14.2:42807): tablet ddfa546f8bd8478380c63e2803d0e153 (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.898625 10322 raft_consensus.cc:2243] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:12:01.898650 3128 webserver.cc:492] Webserver started at http://127.3.14.4:35555/ using document root <none> and password file <none>
I20251026 09:12:01.898715 10322 raft_consensus.cc:2272] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.898723 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:12:01.898802 3128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 09:12:01.898819 10323 ts_tablet_manager.cc:1916] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.898860 3128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 09:12:01.899173 3128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-3-root/instance:
uuid: "79ce812785c0474fbdc50e5e1f122fb3"
format_stamp: "Formatted at 2025-10-26 09:12:01 on dist-test-slave-w8k0"
I20251026 09:12:01.899920 10323 ts_tablet_manager.cc:1929] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 09:12:01.899978 10323 log.cc:1199] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/213642ddd7c749b3baee0f01cd14de97
I20251026 09:12:01.900202 10323 ts_tablet_manager.cc:1950] T 213642ddd7c749b3baee0f01cd14de97 P 50176c38320e477db3a98358734890b2: Deleting consensus metadata
I20251026 09:12:01.900220 10322 ts_tablet_manager.cc:1916] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.900276 3128 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:01.900511 10323 tablet_replica.cc:333] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: stopping tablet replica
I20251026 09:12:01.900568 10323 raft_consensus.cc:2243] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:01.900602 10323 raft_consensus.cc:2272] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.900640 9937 catalog_manager.cc:4985] TS 50176c38320e477db3a98358734890b2 (127.3.14.2:42807): tablet 213642ddd7c749b3baee0f01cd14de97 (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.900822 10323 ts_tablet_manager.cc:1916] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.900942 10333 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.901047 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.901086 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-3-root
uuid: "79ce812785c0474fbdc50e5e1f122fb3"
format_stamp: "Formatted at 2025-10-26 09:12:01 on dist-test-slave-w8k0"
I20251026 09:12:01.901139 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:12:01.901557 10322 ts_tablet_manager.cc:1929] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 09:12:01.901623 10322 log.cc:1199] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/ddfa546f8bd8478380c63e2803d0e153
I20251026 09:12:01.901832 10322 ts_tablet_manager.cc:1950] T ddfa546f8bd8478380c63e2803d0e153 P ac45b5ee4cdb400ba12e12617d0f800a: Deleting consensus metadata
I20251026 09:12:01.902197 10323 ts_tablet_manager.cc:1929] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 09:12:01.902240 10322 tablet_replica.cc:333] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: stopping tablet replica
I20251026 09:12:01.902254 10323 log.cc:1199] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/95cc5ade38624205bad56f670c1ea47d
I20251026 09:12:01.902292 10322 raft_consensus.cc:2243] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:12:01.902346 9937 catalog_manager.cc:4985] TS ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055): tablet ddfa546f8bd8478380c63e2803d0e153 (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.902401 10322 raft_consensus.cc:2272] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.902454 10323 ts_tablet_manager.cc:1950] T 95cc5ade38624205bad56f670c1ea47d P 50176c38320e477db3a98358734890b2: Deleting consensus metadata
I20251026 09:12:01.902797 10323 tablet_replica.cc:333] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: stopping tablet replica
I20251026 09:12:01.902854 10323 raft_consensus.cc:2243] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:01.902882 9937 catalog_manager.cc:4985] TS 50176c38320e477db3a98358734890b2 (127.3.14.2:42807): tablet 95cc5ade38624205bad56f670c1ea47d (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.902889 10323 raft_consensus.cc:2272] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.903049 10323 ts_tablet_manager.cc:1916] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.903553 10322 ts_tablet_manager.cc:1916] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.904089 10323 ts_tablet_manager.cc:1929] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 09:12:01.904141 10323 log.cc:1199] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/8b3912b615dd4121a79577c49631f14a
I20251026 09:12:01.904299 10323 ts_tablet_manager.cc:1950] T 8b3912b615dd4121a79577c49631f14a P 50176c38320e477db3a98358734890b2: Deleting consensus metadata
I20251026 09:12:01.904675 10322 ts_tablet_manager.cc:1929] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 09:12:01.904655 9937 catalog_manager.cc:4985] TS 50176c38320e477db3a98358734890b2 (127.3.14.2:42807): tablet 8b3912b615dd4121a79577c49631f14a (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.904759 10322 log.cc:1199] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/95cc5ade38624205bad56f670c1ea47d
I20251026 09:12:01.904945 10322 ts_tablet_manager.cc:1950] T 95cc5ade38624205bad56f670c1ea47d P ac45b5ee4cdb400ba12e12617d0f800a: Deleting consensus metadata
I20251026 09:12:01.905146 10322 tablet_replica.cc:333] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: stopping tablet replica
I20251026 09:12:01.905195 10322 raft_consensus.cc:2243] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:01.905229 9937 catalog_manager.cc:4985] TS ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055): tablet 95cc5ade38624205bad56f670c1ea47d (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.905231 10322 raft_consensus.cc:2272] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.905400 10322 ts_tablet_manager.cc:1916] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.906425 10322 ts_tablet_manager.cc:1929] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 09:12:01.906481 10322 log.cc:1199] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/8b3912b615dd4121a79577c49631f14a
I20251026 09:12:01.906632 10322 ts_tablet_manager.cc:1950] T 8b3912b615dd4121a79577c49631f14a P ac45b5ee4cdb400ba12e12617d0f800a: Deleting consensus metadata
I20251026 09:12:01.907147 9937 catalog_manager.cc:4985] TS ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055): tablet 8b3912b615dd4121a79577c49631f14a (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.909888 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:12:01.910117 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:12:01.910257 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:12:01.910430 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 09:12:01.910458 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.910492 3128 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 09:12:01.910519 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.913365 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.4:33141
I20251026 09:12:01.913988 3128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 09:12:01.914162 10396 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.4:33141 every 8 connection(s)
W20251026 09:12:01.915520 10400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:12:01.915719 3128 server_base.cc:1047] running on GCE node
W20251026 09:12:01.915691 10402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 09:12:01.915804 10399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 09:12:01.916016 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 09:12:01.916059 3128 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 09:12:01.916076 3128 hybrid_clock.cc:648] HybridClock initialized: now 1761469921916077 us; error 0 us; skew 500 ppm
I20251026 09:12:01.916801 3128 webserver.cc:492] Webserver started at http://127.3.14.1:41759/ using document root <none> and password file <none>
I20251026 09:12:01.916906 3128 fs_manager.cc:362] Metadata directory not provided
I20251026 09:12:01.916947 3128 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 09:12:01.917811 3128 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20251026 09:12:01.918319 10397 heartbeater.cc:344] Connected to a master server at 127.3.14.62:35803
I20251026 09:12:01.918387 10397 heartbeater.cc:461] Registering TS with master...
I20251026 09:12:01.918500 10397 heartbeater.cc:507] Master 127.3.14.62:35803 requested a full tablet report, sending...
I20251026 09:12:01.918756 9949 ts_manager.cc:194] Registered new tserver with Master: 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141)
I20251026 09:12:01.918766 10408 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.918921 3128 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 09:12:01.918975 3128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root
uuid: "54f14d35dc7643e493354830387f763b"
format_stamp: "Formatted at 2025-10-26 09:11:59 on dist-test-slave-w8k0"
I20251026 09:12:01.919034 3128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 09:12:01.919279 9949 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57320
I20251026 09:12:01.930670 3128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 09:12:01.930900 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 09:12:01.931082 3128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 09:12:01.931531 10415 ts_tablet_manager.cc:542] Loading tablet metadata (0/8 complete)
I20251026 09:12:01.933794 3128 ts_tablet_manager.cc:585] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20251026 09:12:01.933866 3128 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20251026 09:12:01.933895 3128 ts_tablet_manager.cc:600] Registering tablets (0/8 complete)
I20251026 09:12:01.934381 10415 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:01.935489 10415 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:01.935667 10415 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Bootstrap complete.
I20251026 09:12:01.935792 10415 ts_tablet_manager.cc:1403] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:01.935940 3128 ts_tablet_manager.cc:616] Registered 8 tablets
I20251026 09:12:01.935925 10415 raft_consensus.cc:359] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.935997 3128 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20251026 09:12:01.936008 10415 raft_consensus.cc:740] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:01.936080 10415 consensus_queue.cc:260] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.936268 10415 ts_tablet_manager.cc:1434] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:12:01.936338 10415 tablet_bootstrap.cc:492] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:01.937263 10415 tablet_bootstrap.cc:492] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:01.937458 10415 tablet_bootstrap.cc:492] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Bootstrap complete.
I20251026 09:12:01.937570 10415 ts_tablet_manager.cc:1403] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:01.937716 10415 raft_consensus.cc:359] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.937819 10415 raft_consensus.cc:740] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:01.937885 10415 consensus_queue.cc:260] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.937981 10415 ts_tablet_manager.cc:1434] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:12:01.938040 10415 tablet_bootstrap.cc:492] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:01.938966 10415 tablet_bootstrap.cc:492] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:01.939160 10415 tablet_bootstrap.cc:492] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Bootstrap complete.
I20251026 09:12:01.939280 10415 ts_tablet_manager.cc:1403] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:01.939445 10415 raft_consensus.cc:359] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.939505 10415 raft_consensus.cc:740] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:01.939568 10415 consensus_queue.cc:260] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.939672 10415 ts_tablet_manager.cc:1434] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.939741 10415 tablet_bootstrap.cc:492] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:01.940055 3128 rpc_server.cc:307] RPC server started. Bound to: 127.3.14.1:39349
I20251026 09:12:01.940136 10476 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.14.1:39349 every 8 connection(s)
I20251026 09:12:01.940740 10415 tablet_bootstrap.cc:492] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:01.940956 10415 tablet_bootstrap.cc:492] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: Bootstrap complete.
I20251026 09:12:01.941097 10415 ts_tablet_manager.cc:1403] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20251026 09:12:01.941232 10415 raft_consensus.cc:359] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.941354 10415 raft_consensus.cc:740] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:01.941437 10415 consensus_queue.cc:260] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.941547 10415 ts_tablet_manager.cc:1434] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.941603 10415 tablet_bootstrap.cc:492] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:01.942821 10415 tablet_bootstrap.cc:492] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 09:12:01.943033 10415 tablet_bootstrap.cc:492] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Bootstrap complete.
I20251026 09:12:01.943114 10415 ts_tablet_manager.cc:1403] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20251026 09:12:01.943264 10415 raft_consensus.cc:359] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.943389 10415 raft_consensus.cc:740] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:01.943478 10415 consensus_queue.cc:260] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.943600 10415 ts_tablet_manager.cc:1434] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:12:01.943677 10415 tablet_bootstrap.cc:492] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:01.943970 10477 heartbeater.cc:344] Connected to a master server at 127.3.14.62:35803
I20251026 09:12:01.944041 10477 heartbeater.cc:461] Registering TS with master...
I20251026 09:12:01.944167 10477 heartbeater.cc:507] Master 127.3.14.62:35803 requested a full tablet report, sending...
I20251026 09:12:01.944571 9949 ts_manager.cc:194] Re-registered known tserver with Master: 54f14d35dc7643e493354830387f763b (127.3.14.1:39349)
I20251026 09:12:01.944687 10415 tablet_bootstrap.cc:492] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:01.944864 10415 tablet_bootstrap.cc:492] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Bootstrap complete.
I20251026 09:12:01.944984 10415 ts_tablet_manager.cc:1403] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:01.945050 9949 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57322
I20251026 09:12:01.945140 10415 raft_consensus.cc:359] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.945217 10415 raft_consensus.cc:740] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:01.945274 10415 consensus_queue.cc:260] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.945379 10415 ts_tablet_manager.cc:1434] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.945436 10415 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:01.945556 10477 heartbeater.cc:499] Master 127.3.14.62:35803 was elected leader, sending a full tablet report...
I20251026 09:12:01.946570 10415 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:01.946725 10415 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: Bootstrap complete.
I20251026 09:12:01.946828 10415 ts_tablet_manager.cc:1403] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:01.947002 10415 raft_consensus.cc:359] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.947054 10415 raft_consensus.cc:740] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:01.947098 10415 consensus_queue.cc:260] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.947217 10415 ts_tablet_manager.cc:1434] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.947279 10415 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: Bootstrap starting.
I20251026 09:12:01.948402 10415 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:01.948539 10415 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: Bootstrap complete.
I20251026 09:12:01.948617 10415 ts_tablet_manager.cc:1403] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:01.948769 10415 raft_consensus.cc:359] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.948827 10415 raft_consensus.cc:740] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f14d35dc7643e493354830387f763b, State: Initialized, Role: FOLLOWER
I20251026 09:12:01.948875 10415 consensus_queue.cc:260] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:01.948985 10415 ts_tablet_manager.cc:1434] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:01.955832 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet ddfa546f8bd8478380c63e2803d0e153 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:56806
I20251026 09:12:01.955832 10439 tablet_service.cc:1552] Processing DeleteTablet for tablet 213642ddd7c749b3baee0f01cd14de97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:56806
I20251026 09:12:01.956135 10486 tablet_replica.cc:333] stopping tablet replica
I20251026 09:12:01.956218 10486 raft_consensus.cc:2243] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:01.956264 10486 raft_consensus.cc:2272] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.956476 10486 ts_tablet_manager.cc:1916] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.958164 10486 ts_tablet_manager.cc:1929] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 09:12:01.958245 10486 log.cc:1199] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/ddfa546f8bd8478380c63e2803d0e153
I20251026 09:12:01.958444 10486 ts_tablet_manager.cc:1950] T ddfa546f8bd8478380c63e2803d0e153 P 54f14d35dc7643e493354830387f763b: Deleting consensus metadata
I20251026 09:12:01.958689 10486 tablet_replica.cc:333] stopping tablet replica
I20251026 09:12:01.958758 10486 raft_consensus.cc:2243] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:01.958768 9934 catalog_manager.cc:4985] TS 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): tablet ddfa546f8bd8478380c63e2803d0e153 (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.958870 10486 raft_consensus.cc:2272] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.958847 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b3912b615dd4121a79577c49631f14a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:56806
I20251026 09:12:01.959116 10486 ts_tablet_manager.cc:1916] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.960440 10486 ts_tablet_manager.cc:1929] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 09:12:01.960510 10486 log.cc:1199] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/213642ddd7c749b3baee0f01cd14de97
I20251026 09:12:01.960665 10486 ts_tablet_manager.cc:1950] T 213642ddd7c749b3baee0f01cd14de97 P 54f14d35dc7643e493354830387f763b: Deleting consensus metadata
I20251026 09:12:01.960938 10486 tablet_replica.cc:333] stopping tablet replica
I20251026 09:12:01.961006 10486 raft_consensus.cc:2243] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:01.961031 10486 pending_rounds.cc:70] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Trying to abort 1 pending ops.
I20251026 09:12:01.961061 10486 pending_rounds.cc:77] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214980795726778368 op_type: NO_OP noop_request { }
I20251026 09:12:01.961097 9934 catalog_manager.cc:4985] TS 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): tablet 213642ddd7c749b3baee0f01cd14de97 (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.961124 10486 raft_consensus.cc:2889] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251026 09:12:01.961158 10486 raft_consensus.cc:2272] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.961395 10486 ts_tablet_manager.cc:1916] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.962670 10486 ts_tablet_manager.cc:1929] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 09:12:01.962728 10486 log.cc:1199] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/8b3912b615dd4121a79577c49631f14a
I20251026 09:12:01.962956 10486 ts_tablet_manager.cc:1950] T 8b3912b615dd4121a79577c49631f14a P 54f14d35dc7643e493354830387f763b: Deleting consensus metadata
I20251026 09:12:01.963371 9934 catalog_manager.cc:4985] TS 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): tablet 8b3912b615dd4121a79577c49631f14a (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:01.981957 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 95cc5ade38624205bad56f670c1ea47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 09:12:01 UTC) from {username='slave'} at 127.0.0.1:56806
I20251026 09:12:01.982146 10486 tablet_replica.cc:333] stopping tablet replica
I20251026 09:12:01.982236 10486 raft_consensus.cc:2243] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:01.982291 10486 raft_consensus.cc:2272] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:01.982583 10486 ts_tablet_manager.cc:1916] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 09:12:01.984063 10486 ts_tablet_manager.cc:1929] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 09:12:01.984128 10486 log.cc:1199] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/95cc5ade38624205bad56f670c1ea47d
I20251026 09:12:01.984269 10486 ts_tablet_manager.cc:1950] T 95cc5ade38624205bad56f670c1ea47d P 54f14d35dc7643e493354830387f763b: Deleting consensus metadata
I20251026 09:12:01.984673 9934 catalog_manager.cc:4985] TS 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): tablet 95cc5ade38624205bad56f670c1ea47d (table dugtrio [id=c51f7950685c44239f6561c30028ccc2]) successfully deleted
I20251026 09:12:02.026387 10129 consensus_queue.cc:237] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:02.027236 10204 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:12:02.027498 10304 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:12:02.028246 10270 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.028326 10204 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.029769 9949 catalog_manager.cc:5649] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a reported cstate change: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.029778 9937 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 36fa073fd46b48eca8f78b1e4f7926f1 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20251026 09:12:02.030375 10450 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:12:02.030625 10304 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 09:12:02.031071 10094 consensus_peers.cc:597] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 -> Peer 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): Couldn't send request to peer 79ce812785c0474fbdc50e5e1f122fb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36fa073fd46b48eca8f78b1e4f7926f1. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:02.031136 10450 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.042156 10204 consensus_queue.cc:237] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:02.042827 10129 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:12:02.043156 10491 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:12:02.044034 10258 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.045193 10129 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.045722 10450 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:12:02.045753 9937 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4059517ece77454a810e8bc9f8a09a9f with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20251026 09:12:02.045964 10274 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 09:12:02.045981 9950 catalog_manager.cc:5649] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a reported cstate change: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:12:02.046435 10450 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
W20251026 09:12:02.046566 10169 consensus_peers.cc:597] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a -> Peer 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): Couldn't send request to peer 79ce812785c0474fbdc50e5e1f122fb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4059517ece77454a810e8bc9f8a09a9f. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:02.084522 10304 consensus_queue.cc:579] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Leader has been unable to successfully communicate with peer 54f14d35dc7643e493354830387f763b for more than 1 seconds (1.508s)
I20251026 09:12:02.085520 10129 consensus_queue.cc:237] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:02.086127 10450 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:12:02.086133 10204 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:12:02.086345 10304 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:12:02.086418 10253 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 09:12:02.086493 10094 consensus_peers.cc:597] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 -> Peer 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): Couldn't send request to peer 79ce812785c0474fbdc50e5e1f122fb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2c3bdaf61b450e8b72ece96f440ee7. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:02.087080 10253 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.087234 10204 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.087226 10450 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.088224 9937 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251026 09:12:02.088433 9949 catalog_manager.cc:5649] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 reported cstate change: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 09:12:02.089526 10129 consensus_queue.cc:237] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:02.089941 10450 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:12:02.089984 10204 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:12:02.090186 10304 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:12:02.090267 10253 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:12:02.090775 10270 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.090938 10450 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.090951 10204 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.091948 9937 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:12:02.092018 9948 catalog_manager.cc:5649] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b reported cstate change: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.095101 10361 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 with delete_type TABLET_DATA_TOMBSTONED (TS 79ce812785c0474fbdc50e5e1f122fb3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35446
W20251026 09:12:02.095567 9936 catalog_manager.cc:4952] TS 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): delete failed for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 because the tablet was not found. No further retry: Not found: Tablet not found: 6b2c3bdaf61b450e8b72ece96f440ee7
I20251026 09:12:02.198381 10274 consensus_queue.cc:579] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Leader has been unable to successfully communicate with peer 54f14d35dc7643e493354830387f763b for more than 1 seconds (1.576s)
I20251026 09:12:02.199501 10204 consensus_queue.cc:237] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:02.200114 10450 raft_consensus.cc:1275] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251026 09:12:02.200161 10169 consensus_peers.cc:597] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a -> Peer 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): Couldn't send request to peer 79ce812785c0474fbdc50e5e1f122fb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2f9f5c2c1e943809cb62f760ad811d6. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:02.200119 10129 raft_consensus.cc:1275] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 09:12:02.200340 10258 consensus_queue.cc:1048] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:12:02.200418 10274 consensus_queue.cc:1048] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 09:12:02.201022 10258 raft_consensus.cc:2955] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.201131 10450 raft_consensus.cc:2955] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.201257 10129 raft_consensus.cc:2955] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.202379 9950 catalog_manager.cc:5649] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b reported cstate change: config changed from index -1 to 2, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:02.202464 9937 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f2f9f5c2c1e943809cb62f760ad811d6 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251026 09:12:02.203367 10204 consensus_queue.cc:237] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:02.203775 10129 raft_consensus.cc:1275] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:12:02.203804 10450 raft_consensus.cc:1275] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:12:02.203969 10491 consensus_queue.cc:1048] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:12:02.204049 10274 consensus_queue.cc:1048] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:12:02.204594 10274 raft_consensus.cc:2955] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.204752 10129 raft_consensus.cc:2955] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.205618 9937 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet f2f9f5c2c1e943809cb62f760ad811d6 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:12:02.205752 9949 catalog_manager.cc:5649] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 reported cstate change: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.205768 10450 raft_consensus.cc:2955] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.206617 10361 tablet_service.cc:1552] Processing DeleteTablet for tablet f2f9f5c2c1e943809cb62f760ad811d6 with delete_type TABLET_DATA_TOMBSTONED (TS 79ce812785c0474fbdc50e5e1f122fb3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35446
W20251026 09:12:02.206843 9936 catalog_manager.cc:4952] TS 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): delete failed for tablet f2f9f5c2c1e943809cb62f760ad811d6 because the tablet was not found. No further retry: Not found: Tablet not found: f2f9f5c2c1e943809cb62f760ad811d6
W20251026 09:12:02.336983 9998 auto_rebalancer.cc:253] failed to send replica move request: Invalid argument: Server with UUID 79ce812785c0474fbdc50e5e1f122fb3 is already a member of the config. RaftConfig: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
W20251026 09:12:02.337478 9998 auto_rebalancer.cc:663] Could not move replica: Illegal state: 36fa073fd46b48eca8f78b1e4f7926f1: source replica ac45b5ee4cdb400ba12e12617d0f800a does not have REPLACE attribute set
W20251026 09:12:02.337530 9998 auto_rebalancer.cc:263] scheduled replica move failed to complete: Illegal state: 36fa073fd46b48eca8f78b1e4f7926f1: source replica ac45b5ee4cdb400ba12e12617d0f800a does not have REPLACE attribute set
W20251026 09:12:02.337925 9998 auto_rebalancer.cc:663] Could not move replica: Illegal state: 4059517ece77454a810e8bc9f8a09a9f: source replica 50176c38320e477db3a98358734890b2 does not have REPLACE attribute set
W20251026 09:12:02.337971 9998 auto_rebalancer.cc:263] scheduled replica move failed to complete: Illegal state: 4059517ece77454a810e8bc9f8a09a9f: source replica 50176c38320e477db3a98358734890b2 does not have REPLACE attribute set
W20251026 09:12:02.338292 9998 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f2f9f5c2c1e943809cb62f760ad811d6, TS 54f14d35dc7643e493354830387f763b -> TS 79ce812785c0474fbdc50e5e1f122fb3 move failed, destination replica disappeared from tablet's Raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
W20251026 09:12:02.338337 9998 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet f2f9f5c2c1e943809cb62f760ad811d6, TS 54f14d35dc7643e493354830387f763b -> TS 79ce812785c0474fbdc50e5e1f122fb3 move failed, destination replica disappeared from tablet's Raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:02.462903 10503 ts_tablet_manager.cc:933] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Initiating tablet copy from peer 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:02.463099 10503 tablet_copy_client.cc:323] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42807
I20251026 09:12:02.465852 10139 tablet_copy_service.cc:140] P 50176c38320e477db3a98358734890b2: Received BeginTabletCopySession request for tablet 36fa073fd46b48eca8f78b1e4f7926f1 from peer 79ce812785c0474fbdc50e5e1f122fb3 ({username='slave'} at 127.0.0.1:54668)
I20251026 09:12:02.465965 10139 tablet_copy_service.cc:161] P 50176c38320e477db3a98358734890b2: Beginning new tablet copy session on tablet 36fa073fd46b48eca8f78b1e4f7926f1 from peer 79ce812785c0474fbdc50e5e1f122fb3 at {username='slave'} at 127.0.0.1:54668: session id = 79ce812785c0474fbdc50e5e1f122fb3-36fa073fd46b48eca8f78b1e4f7926f1
I20251026 09:12:02.466514 10139 tablet_copy_source_session.cc:215] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:12:02.466931 10503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fa073fd46b48eca8f78b1e4f7926f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:02.468075 10503 tablet_copy_client.cc:806] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Starting download of 0 data blocks...
I20251026 09:12:02.468175 10503 tablet_copy_client.cc:670] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Starting download of 1 WAL segments...
I20251026 09:12:02.468763 10503 tablet_copy_client.cc:538] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:12:02.469367 10503 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap starting.
I20251026 09:12:02.470556 10503 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:02.470724 10503 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap complete.
I20251026 09:12:02.470829 10503 ts_tablet_manager.cc:1403] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 09:12:02.470988 10503 raft_consensus.cc:359] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:02.471056 10503 raft_consensus.cc:740] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79ce812785c0474fbdc50e5e1f122fb3, State: Initialized, Role: LEARNER
I20251026 09:12:02.471079 10503 consensus_queue.cc:260] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:02.471249 10503 ts_tablet_manager.cc:1434] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 09:12:02.471307 10397 heartbeater.cc:499] Master 127.3.14.62:35803 was elected leader, sending a full tablet report...
I20251026 09:12:02.471540 10139 tablet_copy_service.cc:342] P 50176c38320e477db3a98358734890b2: Request end of tablet copy session 79ce812785c0474fbdc50e5e1f122fb3-36fa073fd46b48eca8f78b1e4f7926f1 received from {username='slave'} at 127.0.0.1:54668
I20251026 09:12:02.471634 10139 tablet_copy_service.cc:434] P 50176c38320e477db3a98358734890b2: ending tablet copy session 79ce812785c0474fbdc50e5e1f122fb3-36fa073fd46b48eca8f78b1e4f7926f1 on tablet 36fa073fd46b48eca8f78b1e4f7926f1 with peer 79ce812785c0474fbdc50e5e1f122fb3
I20251026 09:12:02.552487 10204 consensus_queue.cc:237] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:02.553002 10450 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:12:02.553011 10129 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:12:02.553221 10274 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:12:02.553330 10491 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:12:02.553941 10258 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.554085 10450 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.554095 10129 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.555197 9937 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 4059517ece77454a810e8bc9f8a09a9f with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:12:02.555245 9948 catalog_manager.cc:5649] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b reported cstate change: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.556160 10361 tablet_service.cc:1552] Processing DeleteTablet for tablet 4059517ece77454a810e8bc9f8a09a9f with delete_type TABLET_DATA_TOMBSTONED (TS 79ce812785c0474fbdc50e5e1f122fb3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35446
W20251026 09:12:02.556373 9936 catalog_manager.cc:4952] TS 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): delete failed for tablet 4059517ece77454a810e8bc9f8a09a9f because the tablet was not found. No further retry: Not found: Tablet not found: 4059517ece77454a810e8bc9f8a09a9f
I20251026 09:12:02.571792 10129 consensus_queue.cc:237] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } }
I20251026 09:12:02.572285 10450 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:12:02.572343 10204 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 09:12:02.572479 10304 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:12:02.572589 10270 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 09:12:02.573194 10253 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.573319 10204 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.573319 10450 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.574424 9937 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 36fa073fd46b48eca8f78b1e4f7926f1 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:12:02.574486 9949 catalog_manager.cc:5649] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b reported cstate change: config changed from index 2 to 3, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) evicted. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } }
I20251026 09:12:02.575461 10361 tablet_service.cc:1552] Processing DeleteTablet for tablet 36fa073fd46b48eca8f78b1e4f7926f1 with delete_type TABLET_DATA_TOMBSTONED (TS 79ce812785c0474fbdc50e5e1f122fb3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35446
I20251026 09:12:02.575587 10497 tablet_replica.cc:333] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: stopping tablet replica
I20251026 09:12:02.575635 10497 raft_consensus.cc:2243] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Raft consensus shutting down.
I20251026 09:12:02.575666 10497 raft_consensus.cc:2272] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Raft consensus is shut down!
I20251026 09:12:02.575892 10497 ts_tablet_manager.cc:1916] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:12:02.577347 10497 ts_tablet_manager.cc:1929] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20251026 09:12:02.577405 10497 log.cc:1199] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-3-root/wals/36fa073fd46b48eca8f78b1e4f7926f1
I20251026 09:12:02.577775 9936 catalog_manager.cc:4985] TS 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): tablet 36fa073fd46b48eca8f78b1e4f7926f1 (table test-workload [id=afb0332ecfa94be5b89d94adb410d780]) successfully deleted
I20251026 09:12:03.339397 10129 consensus_queue.cc:237] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:03.340066 10450 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:12:03.340066 10204 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20251026 09:12:03.340190 10094 consensus_peers.cc:597] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 -> Peer 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): Couldn't send request to peer 79ce812785c0474fbdc50e5e1f122fb3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:03.340317 10508 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:12:03.340438 10304 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:12:03.341150 10508 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.341293 10450 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.341406 10204 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.342571 9950 catalog_manager.cc:5649] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a reported cstate change: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.345995 10204 consensus_queue.cc:237] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:03.346487 10129 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:12:03.346593 10450 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20251026 09:12:03.346652 10169 consensus_peers.cc:597] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a -> Peer 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): Couldn't send request to peer 79ce812785c0474fbdc50e5e1f122fb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4059517ece77454a810e8bc9f8a09a9f. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:03.346738 10516 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:12:03.346848 10274 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:12:03.347509 10274 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.347808 10450 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.348709 10129 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.348812 9948 catalog_manager.cc:5649] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b reported cstate change: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.349197 10128 consensus_queue.cc:237] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:03.349915 10204 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:12:03.349915 10450 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 09:12:03.350154 10508 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 09:12:03.350275 10509 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251026 09:12:03.350432 10094 consensus_peers.cc:597] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 -> Peer 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4:33141): Couldn't send request to peer 79ce812785c0474fbdc50e5e1f122fb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b2c3bdaf61b450e8b72ece96f440ee7. This is attempt 1: this message will repeat every 5th retry.
I20251026 09:12:03.351091 10509 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.351209 10450 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.351482 10204 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.352329 9950 catalog_manager.cc:5649] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b reported cstate change: config changed from index 3 to 4, NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) added. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } } }
I20251026 09:12:03.763302 10521 ts_tablet_manager.cc:933] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Initiating tablet copy from peer 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:03.763602 10521 tablet_copy_client.cc:323] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42807
I20251026 09:12:03.763793 10139 tablet_copy_service.cc:140] P 50176c38320e477db3a98358734890b2: Received BeginTabletCopySession request for tablet 36fa073fd46b48eca8f78b1e4f7926f1 from peer 79ce812785c0474fbdc50e5e1f122fb3 ({username='slave'} at 127.0.0.1:54668)
I20251026 09:12:03.763854 10139 tablet_copy_service.cc:161] P 50176c38320e477db3a98358734890b2: Beginning new tablet copy session on tablet 36fa073fd46b48eca8f78b1e4f7926f1 from peer 79ce812785c0474fbdc50e5e1f122fb3 at {username='slave'} at 127.0.0.1:54668: session id = 79ce812785c0474fbdc50e5e1f122fb3-36fa073fd46b48eca8f78b1e4f7926f1
I20251026 09:12:03.764505 10139 tablet_copy_source_session.cc:215] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:12:03.764762 10521 ts_tablet_manager.cc:1916] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20251026 09:12:03.766566 10521 ts_tablet_manager.cc:1929] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20251026 09:12:03.766680 10521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fa073fd46b48eca8f78b1e4f7926f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:03.767424 10521 tablet_copy_client.cc:806] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Starting download of 0 data blocks...
I20251026 09:12:03.767540 10521 tablet_copy_client.cc:670] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Starting download of 1 WAL segments...
I20251026 09:12:03.768107 10521 tablet_copy_client.cc:538] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:12:03.768800 10521 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap starting.
I20251026 09:12:03.770494 10521 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:03.770704 10521 tablet_bootstrap.cc:492] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap complete.
I20251026 09:12:03.770816 10521 ts_tablet_manager.cc:1403] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20251026 09:12:03.771003 10521 raft_consensus.cc:359] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:03.771112 10521 raft_consensus.cc:740] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79ce812785c0474fbdc50e5e1f122fb3, State: Initialized, Role: LEARNER
I20251026 09:12:03.771152 10521 consensus_queue.cc:260] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:03.771407 10521 ts_tablet_manager.cc:1434] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 09:12:03.771625 10139 tablet_copy_service.cc:342] P 50176c38320e477db3a98358734890b2: Request end of tablet copy session 79ce812785c0474fbdc50e5e1f122fb3-36fa073fd46b48eca8f78b1e4f7926f1 received from {username='slave'} at 127.0.0.1:54668
I20251026 09:12:03.771701 10139 tablet_copy_service.cc:434] P 50176c38320e477db3a98358734890b2: ending tablet copy session 79ce812785c0474fbdc50e5e1f122fb3-36fa073fd46b48eca8f78b1e4f7926f1 on tablet 36fa073fd46b48eca8f78b1e4f7926f1 with peer 79ce812785c0474fbdc50e5e1f122fb3
I20251026 09:12:03.772311 10521 ts_tablet_manager.cc:933] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: Initiating tablet copy from peer ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055)
I20251026 09:12:03.772488 10521 tablet_copy_client.cc:323] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.3:34055
I20251026 09:12:03.775197 10214 tablet_copy_service.cc:140] P ac45b5ee4cdb400ba12e12617d0f800a: Received BeginTabletCopySession request for tablet 4059517ece77454a810e8bc9f8a09a9f from peer 79ce812785c0474fbdc50e5e1f122fb3 ({username='slave'} at 127.0.0.1:51078)
I20251026 09:12:03.775281 10214 tablet_copy_service.cc:161] P ac45b5ee4cdb400ba12e12617d0f800a: Beginning new tablet copy session on tablet 4059517ece77454a810e8bc9f8a09a9f from peer 79ce812785c0474fbdc50e5e1f122fb3 at {username='slave'} at 127.0.0.1:51078: session id = 79ce812785c0474fbdc50e5e1f122fb3-4059517ece77454a810e8bc9f8a09a9f
I20251026 09:12:03.775812 10214 tablet_copy_source_session.cc:215] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:12:03.776209 10521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4059517ece77454a810e8bc9f8a09a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:03.777513 10521 tablet_copy_client.cc:806] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Starting download of 0 data blocks...
I20251026 09:12:03.777650 10521 tablet_copy_client.cc:670] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Starting download of 1 WAL segments...
I20251026 09:12:03.778403 10521 tablet_copy_client.cc:538] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:12:03.779313 10521 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap starting.
I20251026 09:12:03.780944 10521 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:03.781159 10521 tablet_bootstrap.cc:492] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap complete.
I20251026 09:12:03.781265 10521 ts_tablet_manager.cc:1403] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251026 09:12:03.781432 10521 raft_consensus.cc:359] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:03.781527 10521 raft_consensus.cc:740] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79ce812785c0474fbdc50e5e1f122fb3, State: Initialized, Role: LEARNER
I20251026 09:12:03.781575 10521 consensus_queue.cc:260] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:03.781693 10521 ts_tablet_manager.cc:1434] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 09:12:03.781881 10214 tablet_copy_service.cc:342] P ac45b5ee4cdb400ba12e12617d0f800a: Request end of tablet copy session 79ce812785c0474fbdc50e5e1f122fb3-4059517ece77454a810e8bc9f8a09a9f received from {username='slave'} at 127.0.0.1:51078
I20251026 09:12:03.781953 10214 tablet_copy_service.cc:434] P ac45b5ee4cdb400ba12e12617d0f800a: ending tablet copy session 79ce812785c0474fbdc50e5e1f122fb3-4059517ece77454a810e8bc9f8a09a9f on tablet 4059517ece77454a810e8bc9f8a09a9f with peer 79ce812785c0474fbdc50e5e1f122fb3
I20251026 09:12:03.899969 10521 ts_tablet_manager.cc:933] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: Initiating tablet copy from peer 50176c38320e477db3a98358734890b2 (127.3.14.2:42807)
I20251026 09:12:03.900195 10521 tablet_copy_client.cc:323] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.14.2:42807
I20251026 09:12:03.900439 10139 tablet_copy_service.cc:140] P 50176c38320e477db3a98358734890b2: Received BeginTabletCopySession request for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 from peer 79ce812785c0474fbdc50e5e1f122fb3 ({username='slave'} at 127.0.0.1:54668)
I20251026 09:12:03.900520 10139 tablet_copy_service.cc:161] P 50176c38320e477db3a98358734890b2: Beginning new tablet copy session on tablet 6b2c3bdaf61b450e8b72ece96f440ee7 from peer 79ce812785c0474fbdc50e5e1f122fb3 at {username='slave'} at 127.0.0.1:54668: session id = 79ce812785c0474fbdc50e5e1f122fb3-6b2c3bdaf61b450e8b72ece96f440ee7
I20251026 09:12:03.901158 10139 tablet_copy_source_session.cc:215] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 09:12:03.901520 10521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b2c3bdaf61b450e8b72ece96f440ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 09:12:03.902817 10521 tablet_copy_client.cc:806] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Starting download of 0 data blocks...
I20251026 09:12:03.902947 10521 tablet_copy_client.cc:670] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Starting download of 1 WAL segments...
I20251026 09:12:03.903506 10521 tablet_copy_client.cc:538] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 09:12:03.904212 10521 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap starting.
I20251026 09:12:03.905740 10521 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 09:12:03.905939 10521 tablet_bootstrap.cc:492] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: Bootstrap complete.
I20251026 09:12:03.906049 10521 ts_tablet_manager.cc:1403] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 09:12:03.906215 10521 raft_consensus.cc:359] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:03.906283 10521 raft_consensus.cc:740] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79ce812785c0474fbdc50e5e1f122fb3, State: Initialized, Role: LEARNER
I20251026 09:12:03.906320 10521 consensus_queue.cc:260] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: NON_VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: true } }
I20251026 09:12:03.906436 10521 ts_tablet_manager.cc:1434] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 09:12:03.906630 10139 tablet_copy_service.cc:342] P 50176c38320e477db3a98358734890b2: Request end of tablet copy session 79ce812785c0474fbdc50e5e1f122fb3-6b2c3bdaf61b450e8b72ece96f440ee7 received from {username='slave'} at 127.0.0.1:54668
I20251026 09:12:03.906687 10139 tablet_copy_service.cc:434] P 50176c38320e477db3a98358734890b2: ending tablet copy session 79ce812785c0474fbdc50e5e1f122fb3-6b2c3bdaf61b450e8b72ece96f440ee7 on tablet 6b2c3bdaf61b450e8b72ece96f440ee7 with peer 79ce812785c0474fbdc50e5e1f122fb3
I20251026 09:12:04.214632 10371 raft_consensus.cc:1217] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20251026 09:12:04.340332 10371 raft_consensus.cc:1217] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20251026 09:12:04.406442 10371 raft_consensus.cc:1217] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20251026 09:12:04.667198 10532 raft_consensus.cc:1064] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a: attempting to promote NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 to VOTER
I20251026 09:12:04.667505 10532 consensus_queue.cc:237] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } }
I20251026 09:12:04.668030 10129 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 09:12:04.668118 10371 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 09:12:04.668035 10450 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 09:12:04.668308 10536 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 09:12:04.668417 10532 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 09:12:04.668558 10532 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 09:12:04.669332 10517 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.669421 10128 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.670697 9948 catalog_manager.cc:5649] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 reported cstate change: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.670737 10450 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.671151 10371 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } attrs { replace: true } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.673386 10204 consensus_queue.cc:237] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } }
I20251026 09:12:04.673854 10371 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 09:12:04.673830 10450 raft_consensus.cc:1275] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer ac45b5ee4cdb400ba12e12617d0f800a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 09:12:04.674103 10532 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 09:12:04.674201 10531 consensus_queue.cc:1048] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 09:12:04.674680 10532 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 50176c38320e477db3a98358734890b2 (127.3.14.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.674798 10371 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 50176c38320e477db3a98358734890b2 (127.3.14.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.675873 9937 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 4059517ece77454a810e8bc9f8a09a9f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:12:04.675837 10450 raft_consensus.cc:2955] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 50176c38320e477db3a98358734890b2 (127.3.14.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.675904 9950 catalog_manager.cc:5649] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a reported cstate change: config changed from index 5 to 6, VOTER 50176c38320e477db3a98358734890b2 (127.3.14.2) evicted. New cstate: current_term: 1 leader_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:12:04.676869 10118 tablet_service.cc:1552] Processing DeleteTablet for tablet 4059517ece77454a810e8bc9f8a09a9f with delete_type TABLET_DATA_TOMBSTONED (TS 50176c38320e477db3a98358734890b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54650
I20251026 09:12:04.677129 10545 tablet_replica.cc:333] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2: stopping tablet replica
I20251026 09:12:04.677197 10545 raft_consensus.cc:2243] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:04.677284 10545 raft_consensus.cc:2272] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:04.677510 10545 ts_tablet_manager.cc:1916] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:12:04.678969 10545 ts_tablet_manager.cc:1929] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20251026 09:12:04.679028 10545 log.cc:1199] T 4059517ece77454a810e8bc9f8a09a9f P 50176c38320e477db3a98358734890b2: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-1-root/wals/4059517ece77454a810e8bc9f8a09a9f
I20251026 09:12:04.679374 9937 catalog_manager.cc:4985] TS 50176c38320e477db3a98358734890b2 (127.3.14.2:42807): tablet 4059517ece77454a810e8bc9f8a09a9f (table test-workload [id=afb0332ecfa94be5b89d94adb410d780]) successfully deleted
I20251026 09:12:04.744630 10534 raft_consensus.cc:1064] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2: attempting to promote NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 to VOTER
I20251026 09:12:04.744910 10534 consensus_queue.cc:237] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } }
I20251026 09:12:04.745422 10371 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 09:12:04.745505 10450 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 09:12:04.745532 10204 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 09:12:04.745626 10547 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 09:12:04.745782 10533 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 09:12:04.745985 10304 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 09:12:04.746649 10547 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.746829 10204 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.746901 10450 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.748108 9949 catalog_manager.cc:5649] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b reported cstate change: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.748344 10371 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } attrs { replace: true } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.749275 10128 consensus_queue.cc:237] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } }
I20251026 09:12:04.749853 10204 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 09:12:04.749853 10371 raft_consensus.cc:1275] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 09:12:04.750134 10304 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 09:12:04.750321 10551 consensus_queue.cc:1048] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 09:12:04.750850 10304 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 54f14d35dc7643e493354830387f763b (127.3.14.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.751036 10371 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 54f14d35dc7643e493354830387f763b (127.3.14.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.751091 10204 raft_consensus.cc:2955] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 54f14d35dc7643e493354830387f763b (127.3.14.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.751997 9937 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 36fa073fd46b48eca8f78b1e4f7926f1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:12:04.752203 9950 catalog_manager.cc:5649] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 reported cstate change: config changed from index 5 to 6, VOTER 54f14d35dc7643e493354830387f763b (127.3.14.1) evicted. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.753086 10440 tablet_service.cc:1552] Processing DeleteTablet for tablet 36fa073fd46b48eca8f78b1e4f7926f1 with delete_type TABLET_DATA_TOMBSTONED (TS 54f14d35dc7643e493354830387f763b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56806
I20251026 09:12:04.753315 10552 tablet_replica.cc:333] stopping tablet replica
I20251026 09:12:04.753381 10552 raft_consensus.cc:2243] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:04.753515 10552 raft_consensus.cc:2272] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:04.753793 10552 ts_tablet_manager.cc:1916] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:12:04.755115 10552 ts_tablet_manager.cc:1929] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20251026 09:12:04.755172 10552 log.cc:1199] T 36fa073fd46b48eca8f78b1e4f7926f1 P 54f14d35dc7643e493354830387f763b: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-0-root/wals/36fa073fd46b48eca8f78b1e4f7926f1
I20251026 09:12:04.755573 9934 catalog_manager.cc:4985] TS 54f14d35dc7643e493354830387f763b (127.3.14.1:39349): tablet 36fa073fd46b48eca8f78b1e4f7926f1 (table test-workload [id=afb0332ecfa94be5b89d94adb410d780]) successfully deleted
I20251026 09:12:04.811852 10534 raft_consensus.cc:1064] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2: attempting to promote NON_VOTER 79ce812785c0474fbdc50e5e1f122fb3 to VOTER
I20251026 09:12:04.812122 10534 consensus_queue.cc:237] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } }
I20251026 09:12:04.812652 10450 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 09:12:04.812718 10204 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 09:12:04.812856 10304 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 09:12:04.812974 10534 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 09:12:04.813105 10371 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 LEARNER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 09:12:04.813602 10304 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 09:12:04.814235 10534 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.814354 10204 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.814445 10371 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.815423 10450 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.815779 9949 catalog_manager.cc:5649] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 reported cstate change: config changed from index 4 to 5, 79ce812785c0474fbdc50e5e1f122fb3 (127.3.14.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "ac45b5ee4cdb400ba12e12617d0f800a" member_type: VOTER last_known_addr { host: "127.3.14.3" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.816920 10128 consensus_queue.cc:237] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } }
I20251026 09:12:04.817304 10450 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 09:12:04.817317 10371 raft_consensus.cc:1275] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Refusing update from remote peer 50176c38320e477db3a98358734890b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 09:12:04.817467 10304 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 09:12:04.817550 10547 consensus_queue.cc:1048] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 09:12:04.818080 10547 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.818198 10371 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.818374 10450 raft_consensus.cc:2955] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } } }
I20251026 09:12:04.819195 9947 catalog_manager.cc:5649] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 reported cstate change: config changed from index 5 to 6, VOTER ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3) evicted. New cstate: current_term: 1 leader_uuid: "50176c38320e477db3a98358734890b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "54f14d35dc7643e493354830387f763b" member_type: VOTER last_known_addr { host: "127.3.14.1" port: 39349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50176c38320e477db3a98358734890b2" member_type: VOTER last_known_addr { host: "127.3.14.2" port: 42807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ce812785c0474fbdc50e5e1f122fb3" member_type: VOTER last_known_addr { host: "127.3.14.4" port: 33141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 09:12:04.819514 9937 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 09:12:04.820273 10192 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b2c3bdaf61b450e8b72ece96f440ee7 with delete_type TABLET_DATA_TOMBSTONED (TS ac45b5ee4cdb400ba12e12617d0f800a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51034
I20251026 09:12:04.820531 10557 tablet_replica.cc:333] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a: stopping tablet replica
I20251026 09:12:04.820597 10557 raft_consensus.cc:2243] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:04.820667 10557 raft_consensus.cc:2272] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:04.820899 10557 ts_tablet_manager.cc:1916] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 09:12:04.822194 10557 ts_tablet_manager.cc:1929] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20251026 09:12:04.822263 10557 log.cc:1199] T 6b2c3bdaf61b450e8b72ece96f440ee7 P ac45b5ee4cdb400ba12e12617d0f800a: Deleting WAL directory at /tmp/dist-test-taskhlCkFd/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761469742341867-3128-0/minicluster-data/ts-2-root/wals/6b2c3bdaf61b450e8b72ece96f440ee7
I20251026 09:12:04.822674 9937 catalog_manager.cc:4985] TS ac45b5ee4cdb400ba12e12617d0f800a (127.3.14.3:34055): tablet 6b2c3bdaf61b450e8b72ece96f440ee7 (table test-workload [id=afb0332ecfa94be5b89d94adb410d780]) successfully deleted
W20251026 09:12:05.819967 9998 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 50176c38320e477db3a98358734890b2 not available for placement
I20251026 09:12:07.495072 3128 tablet_server.cc:178] TabletServer@127.3.14.1:39349 shutting down...
I20251026 09:12:07.497975 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:12:07.498142 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:12:07.498209 3128 raft_consensus.cc:2243] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:07.498296 3128 raft_consensus.cc:2272] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.498507 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:12:07.498546 3128 raft_consensus.cc:2243] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:07.498596 3128 raft_consensus.cc:2272] T 4059517ece77454a810e8bc9f8a09a9f P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.498746 3128 tablet_replica.cc:333] stopping tablet replica
I20251026 09:12:07.498780 3128 raft_consensus.cc:2243] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:07.498826 3128 raft_consensus.cc:2272] T f2f9f5c2c1e943809cb62f760ad811d6 P 54f14d35dc7643e493354830387f763b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.510735 3128 tablet_server.cc:195] TabletServer@127.3.14.1:39349 shutdown complete.
I20251026 09:12:07.512459 3128 tablet_server.cc:178] TabletServer@127.3.14.2:0 shutting down...
I20251026 09:12:07.515781 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:12:07.516036 3128 tablet_replica.cc:333] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2: stopping tablet replica
I20251026 09:12:07.516132 3128 raft_consensus.cc:2243] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:07.516211 3128 raft_consensus.cc:2272] T f2f9f5c2c1e943809cb62f760ad811d6 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.516418 3128 tablet_replica.cc:333] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2: stopping tablet replica
I20251026 09:12:07.516463 3128 raft_consensus.cc:2243] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:12:07.516537 3128 raft_consensus.cc:2272] T 36fa073fd46b48eca8f78b1e4f7926f1 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.516669 3128 tablet_replica.cc:333] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2: stopping tablet replica
I20251026 09:12:07.516713 3128 raft_consensus.cc:2243] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:12:07.516774 3128 raft_consensus.cc:2272] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 50176c38320e477db3a98358734890b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.528977 3128 tablet_server.cc:195] TabletServer@127.3.14.2:0 shutdown complete.
I20251026 09:12:07.530385 3128 tablet_server.cc:178] TabletServer@127.3.14.3:0 shutting down...
I20251026 09:12:07.534000 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:12:07.534204 3128 tablet_replica.cc:333] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a: stopping tablet replica
I20251026 09:12:07.534291 3128 raft_consensus.cc:2243] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:07.534387 3128 raft_consensus.cc:2272] T 36fa073fd46b48eca8f78b1e4f7926f1 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.534592 3128 tablet_replica.cc:333] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a: stopping tablet replica
I20251026 09:12:07.534642 3128 raft_consensus.cc:2243] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:12:07.534708 3128 raft_consensus.cc:2272] T 4059517ece77454a810e8bc9f8a09a9f P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.534839 3128 tablet_replica.cc:333] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a: stopping tablet replica
I20251026 09:12:07.534878 3128 raft_consensus.cc:2243] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:12:07.534952 3128 raft_consensus.cc:2272] T f2f9f5c2c1e943809cb62f760ad811d6 P ac45b5ee4cdb400ba12e12617d0f800a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.547297 3128 tablet_server.cc:195] TabletServer@127.3.14.3:0 shutdown complete.
I20251026 09:12:07.548832 3128 tablet_server.cc:178] TabletServer@127.3.14.4:0 shutting down...
I20251026 09:12:07.551441 3128 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 09:12:07.551599 3128 tablet_replica.cc:333] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3: stopping tablet replica
I20251026 09:12:07.551666 3128 raft_consensus.cc:2243] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:07.551748 3128 raft_consensus.cc:2272] T 6b2c3bdaf61b450e8b72ece96f440ee7 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.551955 3128 tablet_replica.cc:333] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3: stopping tablet replica
I20251026 09:12:07.552007 3128 raft_consensus.cc:2243] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:07.552062 3128 raft_consensus.cc:2272] T 36fa073fd46b48eca8f78b1e4f7926f1 P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.552186 3128 tablet_replica.cc:333] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3: stopping tablet replica
I20251026 09:12:07.552222 3128 raft_consensus.cc:2243] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 09:12:07.552278 3128 raft_consensus.cc:2272] T 4059517ece77454a810e8bc9f8a09a9f P 79ce812785c0474fbdc50e5e1f122fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.564196 3128 tablet_server.cc:195] TabletServer@127.3.14.4:0 shutdown complete.
I20251026 09:12:07.565354 3128 master.cc:561] Master@127.3.14.62:35803 shutting down...
I20251026 09:12:07.567893 3128 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 1 LEADER]: Raft consensus shutting down.
I20251026 09:12:07.567998 3128 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 09:12:07.568022 3128 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9ba52672e5fe4bbca8dd8cc52e42149f: stopping tablet replica
I20251026 09:12:07.580329 3128 master.cc:583] Master@127.3.14.62:35803 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (8275 ms)
[----------] 14 tests from AutoRebalancerTest (185240 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (185240 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20251026 09:12:07.587159 3128 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5327: suppressed but not reported on 9 messages since previous log ~5 seconds ago
I20251026 09:12:07.587221 3128 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20251026 09:12:07.587252 3128 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20251026 09:12:07.587278 3128 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:4712: suppressed but not reported on 5 messages since previous log ~5 seconds ago
I20251026 09:12:07.587308 3128 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 8 messages since previous log ~171 seconds ago
I20251026 09:12:07.587389 3128 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 51 messages since previous log ~19 seconds ago
I20251026 09:12:07.587482 3128 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 22 messages since previous log ~8 seconds ago